acceptodds
Under review as a conference paper at ICLR 2027

Compressed CP-Conv: Convolutional Learning from Factorized High-Dimensional Inputs

Abstract

Convolutional neural networks conventionally operate on explicitly materialized tensors, making high-resolution and high-dimensional inputs increasingly costly or infeasible to process. We introduce *CP-Conv*, a convolutional operator that applies an arbitrary dense, trainable kernel directly to inputs represented in Canonical Polyadic (CP) form, without reconstructing the dense tensor or factorizing the convolutional kernel. For the tensor represented by the supplied CP factors, CP-Conv is algebraically equivalent to dense convolution and therefore introduces no additional approximation of the operator. Bnecause its output is generally dense, we further introduce *Compressed CP-Conv* (CCP-Conv), which fuses convolution with mode-wise output compression and directly produces a compact representation suitable for standard neural-network layers. We derive the computational and storage complexity of both operators and verify numerical agreement with their dense counterparts. Controlled benchmarks yield operator-level speedups of up to . Across supervised experiments on synthetic data, UCF101, Ego4D, high-dimensional Fokker–Planck problems, and nonlinear shallow-water dynamics from PDEBench, CCP-Conv closely matches matched dense baselines when dense processing is feasible, while substantially reducing training cost. These experiments include both inputs available directly in CP form and dense data represented through approximate CP decomposition. CCP-Conv also enables convolutional learning on inputs whose implicit dense representation contains up to values, well beyond the scale that can be processed densely within the available resources. CCP-Conv integrates into a pretrained R3D-18 architecture with comparable classification performance and supports unsupervised representation learning without dense reconstruction. Together, these results show that exploiting structure directly in the input representation can extend dense-kernel convolutional learning to regimes where materializing the input tensor is computationally prohibitive. Reference implementations in JAX and PyTorch, together with reproduction scripts, are available in the supplementary material.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Tensor Decompositions, Convolutional neural networks, Canonical Polyadic decomposition, efficient deep learning, high-dimensional learning, low-rank representations, tensorized learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.25 over 4 reviews, above 96% of the others.
Reviewer_co3wOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a very interesting idea: instead of compressing the weights, keep the model intact and "compress" the inputs. In addition to this approach, the authors also introduce Compressed CP-Conv (CCP-Conv) in order to prevent producing dense outputs. They test their models on different tasks and models, showing significant speed up and computational efficiency gains.

Critical strengths

The idea is interesting and may prove useful for speeding up expensive workloads, such as training and inference.

Keeping the model intact allows preserving the expressivity of the kernels of the models, in contrast with methods that compress the model. Moreover compressing the inputs and compressing the models could be used in a complementary way.

For training in particular, the cost of decomposing the inputs can be done once and then amortized over different epochs or even on different models trained on the same data. This is a valuable property as one can for instance "compress" large video or scientific datasets once, and use the obtained representation as often as needed.

Critical weaknesses
  1. One of the important weakness, in my view, is the lack of an end to end comparison for data that is not natively in CP form. The authors provide training and inference cost comparison that excludes the cost of the compression of the inputs, which is reported separately in the appendix and that appears high.

For example, a naive rough extrapolation for UCF101 (101 classes):

Cost per sample from your table 12: 10.944

Appendix 1.3 dataset size: 10,160/3,160 training/test samples

Table 3 training time: 200 for ccp, 4200 for dense

This leads to ~145774s, ~40 hours, approximately 34 times more costly than dense training (~1.2 hours).

  1. The comparisons are evaluated on settings where the absolute performance is weak (for example, UCF101 101 cls has 9.5% accuracy. For FP-035, 3 cls, the accuracy of CCP is 38% relatively to 33% chance.) This makes the claim of "comparable to dense" weaker and less informative. In particualr, in the one setting setting with good accuracy (R3D-18, 84% of accuracy), the speedup shrinks to about 5% (not considering input compression cost) and requires a rank of R = 300 (whose time is not reported in Table 12).

  2. The native CP dataset comparison lacks natural baselines. A simple model like a mlp on the factors themselves would have been a relevant baseline.

Questions for authors

Could you provide a end to end wall-clock comparison of the different methods? And provide a discussion on the advantages, drawbacks and practical recommendations.

One of the motivations of this approach is to keep the model weights intact to preserve its expressivity. Could you also provide more insights on the possible drawback, ie. the impact of input fidelity loss by training on compressed inputs? In particular, practical suggestion on how to choose the rank and the impact of large decomposition error?

In Appendix A, you state that "The matched dense baseline deliberately uses the same unstrided convolution and output-compression structure to isolate this difference. More aggressively optimized dense pipelines using striding or pooling could reduce the absolute runtime gap". Could you provide comparisons with the actual optimized pipeline to evaluate if the advantage persists and to evaluate if the gap does not come from this choice?

6 comments
AuthorsOpenReview ↗

Author Response (1/3): CP Decomposition and End-to-End Efficiency

We thank the reviewer for the constructive review and for recognizing the role of input compression, unrestricted kernels, and preprocessing amortization.

Due to the character limit, our response is divided into three parts: (1) CP decomposition and end-to-end costs; (2) predictive performance and factor-native baselines; (3) input fidelity, rank selection, and optimized dense pipelines.

Weakness 1: CP decomposition cost and end-to-end efficiency

We agree with the reviewer that the cost of decomposing dense inputs into CP form must be considered when evaluating the end-to-end computational efficiency of our approach.

We would like to emphasize that our approach is particularly relevant in three situations:

  1. Native CP data: Applications where the data already exist in CP form, eliminating the decomposition cost entirely. This is particularly relevant to high-dimensional scientific applications, such as our Fokker–Planck experiments.

  2. Memory-constrained training: Applications where the dense representation is too large to be processed efficiently on the available hardware. For example, our high-resolution Ego4D experiment involves videos of size 1920 × 2560 × 128, corresponding to approximately 2.34 GiB per video in float32. With a batch size of 20, the dense inputs alone would require approximately 46.9 GiB, before accounting for convolutional outputs, gradients, and intermediate activations. The combined memory requirements make dense training infeasible under our experimental hardware constraints, whereas CCP-Conv allows training directly on the compressed factors.

  3. Repeated training: Applications where the same decomposed dataset is reused across multiple training runs, allowing the one-time decomposition cost to be amortized.

In the first two scenarios, CP-Conv enables computations that would otherwise be infeasible with conventional dense convolution under the available computational resources, whereas the third scenario offers a potential computational advantage through the amortization of preprocessing costs.

To investigate the third scenario, we conducted additional experiments measuring the effect of parallelizing CP decomposition at rank 300, corresponding to the rank used in our R3D-18 experiments.

We benchmarked decomposition with different numbers of parallel CPU workers and evaluated its amortization against the revised stride-matched R3D-18 training times on the complete UCF101 dataset (13,320 videos).

In this experiment, dense R3D-18 training requires 20.73 hours, compared with 15.50 hours for CCP-Conv, corresponding to a saving of approximately 5.23 hours per training run.

The preprocessing costs and corresponding break-even points are:

CPU workersFull UCF101 preprocessing (hours)Training runs to break even
141.648
217.134
410.222
87.952

Break-even values represent repeated complete R3D-18 training runs using cached CP factors.

These results demonstrate that parallelization substantially reduces the preprocessing wall-clock time. With eight workers, the estimated cost of decomposing the complete UCF101 dataset decreases from approximately 41.6 hours to 8 hours.

More importantly, given the approximately 5.23 hours saved per R3D-18 training run, the preprocessing cost can be amortized after only two complete training runs when using eight workers.

We acknowledge that CP decomposition remains computationally expensive and can cost more than the training time saved, particularly when training only once on a dataset that initially exists in dense form.

Nevertheless, decomposition costs do not affect applications with native CP representations, and memory savings can enable training configurations that are infeasible with dense inputs on the available hardware. For deeper architectures such as R3D-18, the substantial savings in training time also make amortization significantly more favorable.

Finally, we emphasize that these break-even estimates are illustrative rather than an exhaustive optimization of the preprocessing pipeline. They combine measured decomposition throughput with training benchmarks and depend on the execution environment and parallelization strategy. Alternative CP decomposition algorithms, optimized implementations, and improved parallelization strategies could further reduce preprocessing costs and change these estimates.

We will include these preprocessing experiments and amortization analyses in the appendix. A systematic investigation of decomposition algorithms, parallelization strategies, and their impact on end-to-end efficiency represents an interesting direction for future work.

AuthorsOpenReview ↗

Author Response (2/3): Predictive Performance and FP Baselines

Weakness 2: Predictive performance and realistic deep architectures

We agree that demonstrating computational speedups on tasks with relatively low classification accuracy is less informative than demonstrating comparable accuracy on more challenging applications.

Accordingly, we conducted additional experiments to strengthen the predictive evaluation.

First, we repeated our R3D-18 experiments on the complete UCF101 dataset using the standard stem stride of (1, 2, 2) for both CCP-Conv and dense convolution using optimized PyTorch code.

Both models were trained for 30 epochs under matched configurations and evaluated across three random seeds.

The results are:

MetricCCP-ConvDense Conv
Test accuracy (%)84.63 ± 0.2485.20 ± 0.79
Training time per epoch (s)1,860 ± 1282,488 ± 43
Total training time (hours)15.5020.73

Under this stride-matched comparison, CCP-Conv achieves comparable classification accuracy (84.63% versus 85.20%), while providing approximately 1.34× faster training, corresponding to a 25.2% reduction in training time.

This demonstrates that the computational advantage persists when comparing models with standard convolution strides, rather than relying on the original unstrided dense baseline.

We acknowledge that the training-time advantage is smaller for this deeper architecture than for the shallow classification experiments. Nevertheless, it remains meaningful while preserving similar predictive performance.

We also conducted additional experiments on high-dimensional native CP data, as described below.

Weakness 3: Stronger factor-native baselines

We agree that the original Fokker–Planck experiments lacked sufficiently strong factor-native baselines.

Accordingly, we expanded the experiments to include linear and RBF Ridge classifiers, log-variance Ridge features, and MLPs trained directly on the CP factors.

Furthermore, we increased the FP-05 dataset from 1,000 to 20,000 samples and adopted a longer training budget.

We also introduced two additional seven-dimensional datasets designed to test more challenging dependencies:

  • FP-HO: A dataset involving high-order interactions.
  • FP-Product: A dataset involving nonlinear products of underlying parameters.

The five-seed test results (mean ± standard deviation, in %) are presented below:

MethodFP-05FP-HOFP-Product
CCP-Conv93.47 ± 3.45100.00 ± 0.0193.80 ± 1.28
Factor MLP99.83 ± 0.0985.17 ± 1.3462.63 ± 0.75
Linear Ridge71.30 ± 0.9175.12 ± 0.3633.92 ± 0.72
RBF Ridge70.86 ± 1.2379.27 ± 0.4747.10 ± 0.58
Log-variance Ridge99.89 ± 0.0433.30 ± 0.8637.51 ± 0.73

We acknowledge that simple statistical features outperform CCP-Conv on the original FP-05 problem.

However, CCP-Conv outperforms all evaluated factor-native baselines on FP-HO and FP-Product, demonstrating that our approach can achieve meaningful predictive performance on more complex high-dimensional problems.

We will include these additional benchmarks and explicitly discuss the limitations of the original FP experiments.

Q1: End-to-end wall-clock comparisons and practical recommendations

We agree that the original distinction between preprocessing costs and model-training times was insufficient for understanding the practical advantages of our approach.

We have addressed this question in Weakness 1, where we provide additional preprocessing benchmarks, estimated end-to-end costs, and amortization calculations.

We will also clarify the three application scenarios where we expect our method to be most relevant: native CP data, memory-constrained training, and repeated training on decomposed datasets.

AuthorsOpenReview ↗

Author Response (3/3): Input Fidelity and Optimized Dense Pipelines

Q2: Input fidelity loss and CP rank selection

We agree that preserving the expressivity of the convolutional kernel does not eliminate approximation errors introduced by compressing dense inputs into CP form.

To investigate the effect of these approximation errors, we conducted rank ablations on UCF101.

In our original R3D-18 rank-ablation experiment, increasing the CP rank improved classification accuracy:

CP rankTest accuracy (%)
10077.0 ± 1.0
22083.0 ± 0.4
30084.8 ± 0.5

These results illustrate the trade-off between input compression and predictive performance. Increasing the rank generally preserves more information but also increases factor storage, computational cost, and preprocessing requirements.

For example, in our UCF101 rank-150 preprocessing experiment, the mean relative reconstruction error was approximately 5.8 ± 2.76%.

In practice, we recommend selecting the CP rank by examining the relative reconstruction error. For each candidate rank, a representative subset of samples can be decomposed, and the lowest rank satisfying a predefined reconstruction-error tolerance can be selected. As a practical starting point, we propose a tolerance of 4-6%, although the appropriate threshold depends on the dataset and downstream task.

Our rank-ablation experiments suggest that classification accuracy can eventually saturate as the CP rank increases. However, the corresponding reconstruction-error threshold is task-dependent, and sometimes relying on validation accuracy as well can be useful.

We have added a detailed discussion of how the CP rank was selected for each experiment to the appendix of the revised paper.

Q3: Comparison with optimized dense pipelines

We agree that the original matched dense baseline was primarily designed to isolate the computational difference between dense convolution and CCP-Conv, rather than represent the fastest possible dense architecture.

Accordingly, we conducted additional comparisons against more optimized dense pipelines, including spatiotemporal resizing, temporal subsampling, patch pooling, and truncated SVD.

For UCF101-10, the revised five-seed results are:

MethodTraining time (s)Test accuracy (%)
CCP-Conv6.6432.72 ± 2.89
Dense Conv155.8632.06 ± 2.84
Spatiotemporal resize28.7232.30 ± 2.14
Temporal subsampling14.6032.24 ± 3.00
Patch pooling28.4833.31 ± 2.69
Truncated SVD157.7631.76 ± 1.42

All methods were evaluated using a consistent PyTorch training implementation, with preprocessing costs excluded from the reported training times.

The results show that more aggressively reduced dense representations can substantially decrease training costs while maintaining similar predictive accuracy.

Nevertheless, CCP-Conv remains approximately 2.20× faster than temporal subsampling, the fastest alternative in this comparison, while achieving comparable classification accuracy.

Furthermore, as discussed in Weakness 2, our additional R3D-18 experiment uses the same standard stride (1, 2, 2) for CCP-Conv and the dense comparator. Under these matched conditions, CCP-Conv retains a 25.2% training-time reduction while achieving similar classification accuracy.

We acknowledge that the advantage of CCP-Conv depends on the architecture and application, and is smaller for some optimized dense pipelines than for the original matched dense baseline.

We will include these comparisons in the revised manuscript and clarify the distinction between operator-level efficiency, training efficiency, and complete end-to-end computational cost.

We thank the reviewer again for their observations, which helped us strengthen the experimental evaluation and clarify the practical advantages and limitations of our approach.

We hope that the additional benchmarks, stronger baselines, rank ablations, and end-to-end cost analysis adequately address the concerns raised in the review.

Reviewer_DCJYOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies convolution on large high-dimensional tensors, for dense convolution becomes too expensive or does not fit in memory. The authors assume the input is given in CP form and propose CP-Conv. It applies an unrestricted trainable kernel directly to the CP factors through a sequence of one-dimensional contractions, without rebuilding the dense tensor, and the result is exact. CCP-Conv additionally fuses learnable mode-wise compression into these contractions, so the output is a small dense tensor that standard layers can use. The experiments check numerical agreement with dense convolution and report runtime scaling. CCP-Conv is then used as a front-end on synthetic videos, UCF101, Ego4D, Fokker–Planck densities and PDEBench shallow water. The paper also plugs it into a pretrained R3D-18 and trains an autoencoder with it. The authors report accuracy close to dense baselines, much shorter training time, and that the method still runs at scales where dense processing is not possible.

Critical strengths
  1. The problem is well motivated. Most tensor work in DL compresses weights, not inputs.
  2. Fusing the compression into the contractions is a intersting practical touch, since the dense output is never formed.
  3. The experiments cover quite different data, from video to PDEs, and the Fokker–Planck case runs at a scale where dense processing is impossible.
  4. The paper is clearly written and fairly honest about its limitations. Code is provided.
Critical weaknesses
  1. The novelty is limited. Eq. 4 is basically Y = Σr K ×1 H_1r ⋯ ×_d H_dr with Hankel matrices built from the factors, and Eq. 6 just puts P_l in front, so both follow from multilinearity. There is also no real related work discussion. Low-rank format convolution (e.g., Kazeev et al., 2013), Tensor Contraction Layers and FFT convolution are not discussed.
  2. The speedups in Table 2 don’t match the paper’s own complexity analysis. For d=3, k=7, R=100, CP-Conv does about 2× more MACs than dense conv but is reported 48× faster, which means the dense baseline runs at only ~1.2 GMAC/s. No hardware or implementation details are given, and there is no FFT baseline.
  3. Training times exclude CP decomposition, but all real datasets start dense. With the numbers in Table 12, CCP is slower overall in most cases, e.g., ~10,800 s vs. 407 s on UCF101-10. The only native-CP data is synthetic.
  4. The experiments don’t say much about learning. The front-end is a single filter followed by linear projections, and accuracies are low (9.5% on UCF101-101). High-res Ego4D has 25 test clips, and Fokker–Planck is only a few points above chance. R3D-18 gains only ~5% in time.
  5. Table 13 looks odd. The SVD baseline goes through the same dense pipeline but trains 4.7× slower than the full-resolution model.
  6. Figure 1 is hard to read.
Questions for authors
  1. What hardware and dense convolution implementation did you use for Table 2? How do you explain CP-Conv being 48× faster while doing about 2× more arithmetic?
  2. More comparation need to do wtih FFT convolution in time and memory.
  3. Why is the SVD baseline in Table 13 slower than the full-resolution dense model, it is counterintuitive.
  4. Please clarify the novelty of the work. Eq. 4 and Eq. 6 can be written as Y = Σr K ×1 H_1r ⋯ ×_d H_dr with Hankel matrices built from the CP factors, which follows directly from multilinearity. What is new beyond this identity, compared with existing work on convolution in low-rank tensor formats (e.g., Rakhuba & Oseledets, 2015; Kazeev et al., 2013) and on trainable mode-wise contractions (e.g., Tensor Contraction Layers, Kossaifi et al., 2017)?
7 comments
AuthorsOpenReview ↗

Author Response (1/4): Novelty and Related Work

We thank the reviewer for taking the time to carefully examine our paper and for providing a detailed and constructive review. We appreciate that the reviewer recognized the motivation behind our approach, the computational benefits of fused compression, and the potential for processing high-dimensional data that would be infeasible to handle in dense form.

We agree with several of the limitations highlighted by the reviewer, particularly concerning the discussion of related work, the original benchmarking methodology, and the need for stronger experimental evidence.

We are pleased to share that we conducted multiple additional experiments to address these concerns, including optimized convolution benchmarks, improved training-time comparisons, stronger factor-native baselines, and additional experiments on deep video architectures.

We hope that these additional experiments and clarifications adequately address the reviewer's concerns and provide a stronger basis for reassessing the paper.

Due to the character limit, our response is divided into four comments: (1) novelty and related work; (2) optimized benchmarks and SVD timing; (3) CP decomposition and amortization; (4) predictive performance and factor-native baselines.

Weakness 1 and Q4: Novelty and relationship to existing tensor methods

We thank the reviewer for highlighting these references. We agree that the algebraic identities underlying Eqs. (4) and (6) follow from multilinearity. However, we would like to clarify that the cited works address substantially different problems from the one investigated in our paper.

Previous works on convolution in low-rank tensor formats, such as Kazeev et al. (2013) and Rakhuba and Oseledets (2015), primarily focus on efficient numerical computation using tensor-structured representations and approximation algorithms, rather than trainable convolutional operators for machine learning. On the other hand, Kossaifi et al. (2017) introduce Tensor Contraction Layers that perform learnable mode-wise projections on already materialized activation tensors. In contrast, CCP-Conv integrates these projections directly into convolution on CP factors, avoiding materialization of both the dense input and the full convolutional output. While these works share relevant mathematical tools with ours, none investigates the specific combination of unrestricted trainable convolution kernels on CP inputs and fused output compression.

More precisely, our contributions can be summarized in three points:

  1. Unrestricted trainable convolution kernels: CP-Conv exploits the low-rank structure of the input without imposing low-rank or separability constraints on the convolution kernel. This preserves the expressivity of an arbitrary dense kernel while operating directly on CP factors.

  2. Fused convolution and compression: CCP-Conv incorporates learnable mode-wise output projections directly into the convolutional contractions. Consequently, neither the dense input nor the full convolution output needs to be materialized, which is particularly important in high-dimensional settings.

  3. Integration into trainable architectures: We provide differentiable JAX and PyTorch implementations, allowing convolution kernels and compression projections to be learned jointly with downstream networks. We demonstrate this approach in shallow classifiers, deep video architectures such as R3D-18, and unsupervised representation learning.

We acknowledge that the underlying multilinear identities are straightforward consequences of tensor algebra. However, our contribution lies in the formulation, efficient implementation, and empirical evaluation of a trainable convolutional operator combining these properties, rather than in introducing a new tensor identity.

We have added a discussion of these related works and clarified the distinctions from our approach in the introduction of the revised manuscript.

AuthorsOpenReview ↗

Author Response (2/4): Benchmarks and SVD Timing

Weakness 2 and Q1–Q2: Computational complexity and optimized convolution baselines

The original Table 2 used a suboptimal JAX CPU convolution baseline.

We reran Table 2 on the workstation used for Tables 3 and 13, comparing CP-Conv and CCP-Conv against optimized PyTorch dense baselines. We also tested cuDNN and reconstruct–FFT-convolve–project on our local GPU.

Below are median forward times (ms; 20 post-warmup runs), excluding CP decomposition. Unless stated otherwise: R=100, k=7, m=16.

CPU benchmarks (local Intel Core Ultra 9 275HX, PyTorch and FFT)

dNPyTorch Conv + projectionCCP-Conv
21280.1570.362
3641.7350.666
312818.0410.781
316039.5500.817

Each factorized operator is compared with its corresponding dense baseline.

At d=3, N=128, k=7, R=100, m=16, CP-Conv achieves 2.04× speedup and CCP-Conv achieves 23.10×. These replace our earlier, overly optimistic claims.

GPU benchmarks (NVIDIA RTX 5080 Laptop GPU, cuDNN and FFT)

dNcuDNN Conv + projectionFFT Conv + projectionCCP-Conv
21280.1740.3440.552
3640.2980.6120.914
31281.0380.6440.953
31602.0150.5720.787
4128N/A60.451.48

N/A indicates that cuDNN does not support four spatial dimensions.

Wall-clock performance also depends on hardware and implementation.

The revised benchmarks show that optimized FFT and cuDNN implementations can outperform CCP-Conv in certain lower-dimensional settings. Similarly, the CPU results show that CP-Conv is not uniformly faster than optimized dense convolution. We acknowledge these results and do not claim that our approach is universally faster than optimized dense alternatives.

However, the advantage of CCP-Conv becomes substantial in higher-dimensional settings, where dense intermediate tensors become increasingly expensive. For example, at d=4, N=128, k=7, R=100, m=16, CCP-Conv achieves a 40.8× speedup over FFT convolution followed by projection (1.48 ms versus 60.45 ms).

Furthermore, the FFT-based pipeline requires approximately 4,144 MiB of incremental peak GPU memory, compared with 14.8 MiB for CCP-Conv.

These results demonstrate that the computational advantage of CCP-Conv depends on dimensionality, compression, and the cost of materializing intermediate tensors, with particularly significant benefits in higher-dimensional applications.

The revised manuscript will report the complete local CPU parameter sweeps in Table 2, including both CP-Conv and CCP-Conv. The additional GPU comparisons against cuDNN and FFT-based alternatives, together with hardware specifications, benchmarking methodology, and operation-count analyses, will be included in the manuscript and appendix as appropriate.

Weakness 5 and Q3: SVD baseline inconsistency in Table 13

We thank the reviewer for identifying the unexpected discrepancy between the dense-convolution and SVD training times.

Following this observation, we investigated the original experiments and found that differences in execution environments and implementation conditions had affected the comparison.

Accordingly, we repeated the UCF101-10 experiments on our local Intel Core Ultra 9 275HX workstation, using a consistent PyTorch implementation, timing methodology, and five random seeds.

The revised results are:

MethodTraining time (s)
CCP-Conv6.64
Dense convolution155.86
Spatiotemporal resize28.72
Temporal subsampling14.60
Patch pooling28.48
Truncated SVD157.76

All methods use the same training pipeline, with one-time preprocessing costs excluded from the reported training times.

The revised dense-convolution and SVD timings are now consistent: 155.86 s versus 157.76 s, rather than the originally reported 407 s versus 1,906 s.

Furthermore, the CCP-Conv and dense-convolution results in Tables 3 and 13 now correspond to exactly the same measurements, eliminating the previous inconsistency.

We also observe that optimized dense-input reduction methods can substantially reduce training time. Nevertheless, CCP-Conv remains approximately 2.20× faster than temporal subsampling, the fastest alternative in this comparison, while achieving comparable classification accuracy.

We will replace the previous timing results and specify the hardware and benchmarking methodology used.

AuthorsOpenReview ↗

Author Response (3/4): CP Decomposition and Amortization

Weakness 3: CP decomposition cost and end-to-end efficiency

We agree with the reviewer that the cost of decomposing dense inputs into CP form must be considered when evaluating the end-to-end computational efficiency of our approach.

We would like to emphasize that our approach is particularly relevant in three situations:

  1. Native CP data: Applications where the data already exist in CP form, eliminating the decomposition cost entirely. This is particularly relevant to high-dimensional scientific applications, such as our Fokker–Planck experiments.

  2. Memory-constrained training: Applications where the dense representation is too large to be processed efficiently on the available hardware. For example, our high-resolution Ego4D experiment involves videos of size 1920 × 2560 × 128, corresponding to approximately 2.34 GiB per video in float32. With a batch size of 20, the dense inputs alone would require approximately 46.9 GiB, before accounting for convolutional outputs, gradients, and intermediate activations. This makes dense training infeasible under our experimental hardware constraints, whereas CCP-Conv allows training directly on the compressed factors.

  3. Repeated training: Applications where the same decomposed dataset is reused across multiple training runs, allowing the one-time preprocessing cost to be amortized.

In the first two scenarios, CP-Conv enables computations that would otherwise be infeasible with conventional dense convolution under the available computational resources, whereas the third scenario offers a potential computational advantage through the amortization of preprocessing costs.

To investigate the third scenario, we conducted additional experiments measuring the effect of parallelizing CP decomposition at rank 300, corresponding to the rank used in our R3D-18 experiments.

We benchmarked decomposition on 128 UCF101 videos using different numbers of parallel CPU workers and estimated the corresponding preprocessing costs for the complete UCF101 dataset (13,320 videos).

Using our revised stride-matched R3D-18 training measurements, dense convolution requires 20.73 hours, compared with 15.50 hours for CCP-Conv, corresponding to a saving of approximately 5.23 hours per training run.

The preprocessing costs and corresponding break-even points are:

CPU workersEstimated full UCF101 preprocessing (hours)Training runs to break even
141.648
217.134
410.222
87.952

Break-even values represent repeated complete R3D-18 training runs using cached CP factors.

These results demonstrate that parallelization substantially reduces the preprocessing wall-clock time. With eight workers, the estimated cost of decomposing the complete UCF101 dataset decreases from approximately 41.6 hours to 8 hours.

More importantly, given the approximately 5.23 hours saved per R3D-18 training run, the preprocessing cost can be amortized after approximately two complete training runs when using eight workers.

We acknowledge that CP decomposition remains computationally expensive and can cost more than the training time saved, particularly when training only once on a dataset that initially exists in dense form. Furthermore, new dense inputs must also be decomposed before inference.

These break-even values are illustrative estimates combining preprocessing and training measurements, and depend on the hardware and execution conditions. Alternative CP decomposition algorithms, optimized implementations, and improved parallelization strategies could further reduce preprocessing costs.

We will include these preprocessing experiments and amortization analyses in the appendix, and clarify the distinction between preprocessing costs, training efficiency, and memory feasibility.

Reviewer_tsKiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper applies a dense, trainable convolution kernel directly to CP factors by contracting the kernel mode by mode with sliding windows of each factor (CP-Conv, exact). CCP-Conv also folds mode-wise projections into those windows, so neither the dense input nor the full convolution output is ever formed. The experiments report:

  • operator speedups up to 3.9×10³;
  • accuracy parity with a matched dense pipeline at 10-156x lower training time on synthetic video, UCF101, Ego4D and PDEBench;
  • training on 7-D Fokker-Planck inputs with 2.94×10¹¹ implicit entries;
  • a CCP stem in R3D-18 (84.8% vs 84.2% dense);
  • a CP-native autoencoder.
Critical strengths

S1. The operator is exact, and the complexity analysis checks out. The fused compression in Eq. (6) is the valuable part. Its cost has no Nᵈ term, so data that already arrive in CP form can get a learned convolutional front-end at a cost set by the rank rather than the grid. The selection-matrix initialization of the R3D-18 stem is a nice touch.

S2. The supplement is complete enough to audit: JAX and PyTorch code, tests, configs that match Table 7, and data generators. The paper is also candid in places (A.4, A.11).

Critical weaknesses

W1. The timing evidence, which is the paper's main claim, doesn't hold up as presented.

(a) Table 2 contradicts the authors' own cost proxy (cp_conv_cost_exact). At d=3, N=128, k=7, R=100, CP-Conv does 2.2x more arithmetic than dense convolution, yet it is reported 48x faster. For CCP-Conv the arithmetic ratio is about 71x, against 607x reported. The dense path runs at roughly 1.2 GMAC/s. A single-threaded NumPy FFT convolution on my laptop takes about 90 ms for the case reported at 502 ms. The paper has no FFT baseline and doesn't state its hardware.

(b) The SVD row in Table 13 trains the same dense model as the Table 3 dense UCF101-10 run, yet takes 1906 s instead of 407 s. The CCP row in Table 13 is copied from Table 3. At Table 3's rate, temporal subsampling would take about 30 s, faster than CCP (39.7 s) at similar accuracy.

(c) Normalized per output voxel, dense time per step varies about 30x across Table 3. The 156x Ego4D speedup comes from the outlier row.

(d) The R3D-18 dense comparator uses a stride-1 stem, about 4x the compute of the pretrained stride-(1,2,2) stem.

(e) Decomposing UCF101 takes about 40 h of compute, against about 1.1 h of training saved. Every new clip must also be decomposed at inference.

Since speed is the central argument, I can't currently tell how much of the gain comes from the method.

W2. The learning results don't show value where the method is meant to help.

  • Parity with the matched dense baseline is expected by construction. A.9 and A.11 show the shallow tasks are insensitive to rank and to simple input reductions.
  • At scale, results are near chance on small test sets. Ego4D-high gets 29% on 25 clips (20% chance, p≈0.22), and FP-035 gets 38% (33% chance, p≈0.09).
  • The FP task is also easy. On data from the released generator, a ridge classifier on the logs of six marginal variances, computed exactly from the factors, reaches 99.8% (FP-05) and 98.1% (FP-035). CCP-Conv gets 41% and 38% after only 100 training steps.
  • The 61-day dense-autoencoder projection reflects a decoder that reconstructs the full video (about 5.7×10⁹ MACs per sample). It says little about the input convolution that CCP replaces (about 1.2×10⁷).

Minor: Eq. (4) is a sum of Tucker tensors with core K and Hankel factor matrices. Prior convolution work in CP/Tucker formats (Khoromskij, 2010; Kazeev et al., 2013) and tensor contraction layers (Kossaifi et al., 2017) deserve discussion.

Questions for authors
  1. (Clarification) What hardware was used? Why does the same dense computation take 407 s in Table 3 and 1906 s in Table 13? Please re-time both tables in one environment.
  2. (Experiment) Please add FFT and cuDNN dense baselines to Table 2, with operation counts. At the video settings, please also compare CCP-Conv with reconstructing, convolving by FFT, then projecting.
  3. (Experiment) Please add factor-native FP baselines, and an ablation with K fixed to a delta to test whether the dense kernel matters.
  4. (Clarification) Is the R3D-18 time per epoch or total? How does the unmodified stride-(1,2,2) model compare?

I'd raise my score given consistent timings against stronger baselines, plus one at-scale task where CCP-Conv beats simple factor-native alternatives.

9 comments
AuthorsOpenReview ↗

Author Response (1/4): Optimized Operator Benchmarks

We thank the reviewer for taking the time to thoroughly examine our paper, code, and experimental methodology, and for providing a detailed and constructive review. We appreciate that the reviewer recognized the exactness of CP-Conv, the computational benefits of fused compression in CCP-Conv, and the potential of our approach for learning directly from high-dimensional CP representations.

We agree with several of the limitations highlighted by the reviewer, and we are pleased to share that we conducted multiple additional experiments to address these concerns.

We hope that these additional experiments and clarifications adequately address the reviewer's concerns and provide a stronger basis for reassessing the paper.

Due to the 5,000-character limit, we respond in four parts: (1) operator benchmarks; (2) training consistency, R3D-18, and autoencoder; (3) CP decomposition costs; (4) factor-native baselines, kernel ablations, and related work.

Weakness 1(a): Computational complexity and optimized convolution baselines

Prompted by the reviewer's observation, we discovered that the original experiments in Table 2 used a JAX convolution implementation that was not well optimized for CPU execution.

Accordingly, we repeated the complete Table 2 parameter sweeps on our local workstation using optimized PyTorch dense convolution, and compared both CP-Conv and CCP-Conv against their corresponding dense implementations. This is the same workstation used for the revised supervised-learning benchmarks in Tables 3 and 13. We additionally performed separate GPU comparisons against optimized cuDNN and FFT-based convolution, including a reconstruct–FFT-convolve–project pipeline.

We present representative results below. All configurations use CP rank R=100, kernel size k=7, and compressed output size m=16 along each dimension, unless otherwise specified. The tables report median forward-execution times in milliseconds over 20 post-warmup repetitions, excluding CP decomposition and input preprocessing.

CPU benchmarks (local Intel Core Ultra 9 275HX, PyTorch and FFT)

dNPyTorch Conv + projectionFFT Conv + projectionCCP-Conv
21280.1575.3450.362
3641.73513.3960.666
312818.04118.3360.781
316039.55037.4990.817

CP-Conv is compared against conventional dense convolution, while CCP-Conv is compared against dense convolution followed by the same mode-wise projection.

For the configuration highlighted by the reviewer (d=3, N=128, k=7, R=100, m=16), CCP-Conv achieves a 23.10× speedup over dense convolution followed by projection.

We note that CP-Conv is not uniformly faster than optimized dense convolution in lower-dimensional settings. The computational advantage depends on the input dimensionality, CP rank, output compression, and implementation.

GPU benchmarks (NVIDIA RTX 5080 Laptop GPU, cuDNN and FFT)

dNcuDNN Conv + projectionFFT Conv + projectionCCP-Conv
21280.1740.3440.552
3640.2980.6120.914
31281.0380.6440.953
31602.0150.5720.787
4128N/A60.451.48

N/A indicates that cuDNN does not support four spatial dimensions.

We note that theoretical complexity characterizes arithmetic scaling but does not directly predict wall-clock performance, which also depends on implementation, memory access, and hardware.

The GPU results confirm that optimized FFT and cuDNN implementations can outperform CCP-Conv in certain lower-dimensional settings. We acknowledge this limitation and do not claim that our approach is universally faster than optimized dense alternatives.

However, the advantage of CCP-Conv becomes substantial in higher-dimensional settings, where dense intermediate tensors become increasingly expensive. For example, at d=4, N=128, k=7, R=100, m=16, CCP-Conv achieves a 40.8× speedup over FFT convolution followed by projection (1.48 ms versus 60.45 ms). Furthermore, the FFT-based pipeline requires approximately 4,144 MiB of incremental peak GPU memory, compared with 14.8 MiB for CCP-Conv.

These results demonstrate that CCP-Conv is particularly advantageous when output compression avoids the materialization of large dense intermediate tensors, rather than in every convolutional configuration.

We have updated Table 2 with optimized PyTorch CPU baselines, operation counts, and benchmarking details. Additional comparisons against FFT-based convolution on the local CPU and cuDNN and FFT-based convolution on the GPU are reported separately in the appendix.

AuthorsOpenReview ↗

Author Response (2/4): Training Consistency and R3D-18

Weakness 1(b–c): Inconsistent timings in Tables 3 and 13

We thank the reviewer for identifying the inconsistencies between Tables 3 and 13, as well as the unexpectedly large dense-convolution training time on Ego4D.

Following these observations, we investigated the original experiments and found differences in execution environments and limitations in the original JAX dense-convolution implementation.

Accordingly, we repeated the supervised classification benchmarks using a consistent PyTorch implementation and five seeds on the same CPU.

For the UCF101-10 experiment, the revised training times are:

MethodTraining time (s)
CCP-Conv6.64
PyTorch dense convolution155.86
Spatiotemporal resize28.72
Temporal subsampling14.60
Patch pooling28.48
Truncated SVD157.76

All methods use the same training pipeline, and the one-time preprocessing costs are excluded from these measurements.

The revised dense-convolution and SVD timings are now consistent. CCP-Conv is also approximately 2.20× faster than temporal subsampling in this experiment.

Furthermore, Tables 3 and 13 now use the exact same CCP-Conv and dense-convolution measurements for UCF101-10, eliminating the previous inconsistency.

Regarding the Ego4D outlier, the revised dense-convolution training time is 243.60 s, compared with 155.86 s for UCF101-10. Their ratio is now approximately 1.56×, rather than the originally reported 15.3×.

Both experiments use 500 optimization steps, and Ego4D has a temporal dimension 1.5× larger than UCF101.

These revised measurements resolve the previously unexplained timing discrepancy. We will replace the earlier results and specify the hardware and experimental conditions used.

Weakness 1(d): R3D-18 stride configuration

We agree that using a stride-1 dense comparator introduces an architectural difference that could affect the computational comparison.

Accordingly, we conducted an additional experiment on the complete UCF101 dataset using the standard R3D-18 stem stride of (1, 2, 2) for both CCP-Conv and dense convolution.

Both models were trained for 30 epochs using matched configurations and evaluated across three random seeds.

The results are:

MetricCCP-ConvDense Conv
Test accuracy (%)84.63 ± 0.2485.20 ± 0.79
Training time per epoch (s)1,860 ± 1282,488 ± 43
Total training time (hours)15.5020.73

Under this stride-matched comparison, CCP-Conv achieves approximately 1.34× faster training, corresponding to a 25.2% reduction in training time.

We will include these results in the revised paper.

Regarding the timing convention in the original Table 5, we will clarify the distinction between per-epoch training time and complete training time.

Additional concern: Dense-autoencoder training-time extrapolation

We agree with the reviewer that the original 61-day dense-autoencoder extrapolation is not an appropriate measurement of the computational advantage attributable specifically to CCP-Conv.

As the reviewer correctly pointed out, the dense autoencoder reconstructs the complete video, whereas our CP-native decoder predicts factors and evaluates the reconstruction objective without materializing the full dense output.

Consequently, the reported difference includes computational costs associated with different decoder architectures.

We will therefore remove the projected dense-autoencoder training-time comparison from the paper.

However, we will retain the unsupervised representation-learning experiment, which demonstrates that CCP-Conv can support reconstruction-based learning directly in CP form without requiring class labels.

Q1 and Q4: Hardware and R3D-18 timing clarification

The revised Tables 3 and 13 use a consistent PyTorch implementation on an Intel Core Ultra 9 275HX CPU (24 cores, 24 logical processors); the GPU operator benchmarks use an NVIDIA RTX 5080 Laptop GPU. We will specify hardware and timing conventions in the revised paper and distinguish R3D-18 per-epoch times from cumulative training times, as reported above.

AuthorsOpenReview ↗

Author Response (3/4): CP Decomposition and End-to-End Efficiency

Weakness 1(e): CP decomposition cost and end-to-end efficiency

We agree with the reviewer that the cost of decomposing dense inputs into CP form must be considered when evaluating the end-to-end computational efficiency of our approach.

We would like to emphasize that our approach is particularly relevant in three situations:

  1. Native CP data: Applications where the data already exist in CP form, eliminating the decomposition cost entirely. This is particularly relevant to high-dimensional scientific applications, such as our Fokker–Planck experiments.

  2. Memory-constrained training: Applications where the dense representation is too large to be processed efficiently on the available hardware. For example, our high-resolution Ego4D experiment involves videos of size 1920 × 2560 × 128, corresponding to approximately 2.34 GiB per video in float32. With a batch size of 20, the dense inputs alone would require approximately 46.9 GiB, before accounting for convolutional outputs, gradients, and intermediate activations. The combined memory requirements make dense training infeasible under our experimental hardware constraints, whereas CCP-Conv allows training directly on the compressed factors.

  3. Repeated training: Applications where the same decomposed dataset is reused across multiple training runs, allowing the one-time decomposition cost to be amortized.

In the first two scenarios, CP-Conv enables computations that would otherwise be infeasible with conventional dense convolution under the available computational resources, whereas the third scenario offers a potential computational advantage through the amortization of preprocessing costs.

To investigate the third scenario, we conducted additional experiments measuring the effect of parallelizing CP decomposition at rank 300, corresponding to the rank used in our R3D-18 experiments.

We benchmarked decomposition with different numbers of parallel CPU workers and evaluated its amortization against the revised stride-matched R3D-18 training times on the complete UCF101 dataset (13,320 videos).

In this experiment, dense R3D-18 training requires 20.73 hours, compared with 15.50 hours for CCP-Conv, corresponding to a saving of approximately 5.23 hours per training run.

The preprocessing costs and corresponding break-even points are:

CPU workersFull UCF101 preprocessing (hours)Training runs to break even
141.648
217.134
410.222
87.952

Break-even values represent repeated complete R3D-18 training runs using cached CP factors.

These results demonstrate that parallelization substantially reduces the preprocessing wall-clock time. With eight workers, the estimated cost of decomposing the complete UCF101 dataset decreases from approximately 41.6 hours to 8 hours.

More importantly, given the approximately 5.23 hours saved per R3D-18 training run, the preprocessing cost can be amortized after only two complete training runs when using eight workers.

We acknowledge that CP decomposition remains computationally expensive and can cost more than the training time saved, particularly when training only once on a dataset that initially exists in dense form.

Nevertheless, decomposition costs do not affect applications with native CP representations, and memory savings can enable training configurations that are infeasible with dense inputs on the available hardware. For deeper architectures such as R3D-18, the substantial savings in training time also make amortization significantly more favorable.

Finally, we emphasize that these break-even estimates are illustrative rather than an exhaustive optimization of the preprocessing pipeline. They combine measured decomposition throughput with training benchmarks and depend on the execution environment and parallelization strategy. Alternative CP decomposition algorithms, optimized implementations, and improved parallelization strategies could further reduce preprocessing costs and change these estimates.

We will include these preprocessing experiments and amortization analyses in the appendix. A systematic investigation of decomposition algorithms, parallelization strategies, and their impact on end-to-end efficiency represents an interesting direction for future work.

Reviewer_QLfmOpenReview ↗
Rating
4(+2)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

CP decomposition reduces the memory needed for storage by representing a high-dimensional tensor as a sum of rank-1 tensors and storing it. By presenting CP-Conv, a method for convolution between an input tensor given in CP form and a general dense kernel, this paper claims that computation and training can be done directly without converting the CP-form input to dense, while keeping the kernel in dense form without decomposing or constraining its form. In addition, it presents CCP-Conv, which merges the step of applying axis-wise linear compression to the output into the operation, obtaining a small compressed tensor without forming the output in dense form either. As a result, it shows that training is possible even on inputs for which training was impossible at that scale in dense form.

Critical strengths
  1. When the input is in CP form, CP-Conv and CCP-Conv obtain results exactly identical to dense convolution (and its compressed result), while not constraining the form of the kernel. This is clean in that, unlike prior work that constrained the form of the kernel, it reduces computation using only the structure on the input side.
  2. By merging the output compression inside the operation, CCP-Conv makes convolution-based training possible on inputs that cannot be loaded into memory in dense form.
Critical weaknesses
  1. In Table 2, when d = 3, k = 7, R = 100, N = 128, according to the cost analysis in the paper, CP-Conv should be roughly about 2 times slower than dense, but the measured runtime is about 48 times faster. As such, the measured speedup does not match the cost analysis in the paper, and the hardware used in the experiments is also not specified. It seems that the speed difference measured in the experiments arises not from the efficiency of the algorithm itself but from the inefficiency of the convolution library in a single-filter setting.
  2. The comparison targets are not appropriate. The paper claims that for native CP data the methods can be appropriately utilized because no CP decomposition cost is incurred, and to show that CCP-Conv is useful on native CP data such as FP-05 and FP-035, which are examples of this, an appropriate baseline (dense convolution on a coarse grid, etc.) is needed. Since the current paper has no baseline, it cannot be known whether CCP-Conv’s 41 and 38% (chance 33.3%) are meaningful performance on this task. The current experiments show that training at that scale is feasible, but they do not show that the results are useful. Also, to be persuasive, the paper needs to present a situation where a dense kernel is actually needed, which is the situation where the strengths of this paper come into play. It seems that the benefit of the work would be clearly convincing if the paper compared a setting applying a dense kernel and CCP-Conv (or CP-Conv) with a setting using a kernel from prior work constrained to a specific form (separable kernels, etc.), and presented a situation in which an advantage is obtained.
Questions for authors
  1. What hardware was used in the experiments?
  2. According to Table 3, on Ego4D Low res the training time of dense is about 15 times longer than on UCF101-10, whose input size is only 1.5 times smaller. By my calculation the number of steps seems similar; what is the reason for the difference in training time?
10 comments
AuthorsOpenReview ↗

Author Response (1/2): Benchmarks and Timing Clarifications

We thank the reviewer for taking the time to understand our paper and for providing a thoughtful and high-quality review. We appreciate that the reviewer recognized the exactness of CP-Conv with unrestricted dense kernels and the ability of CCP-Conv to avoid materializing high-dimensional outputs.

We agree with both limitations highlighted by the reviewer, and we are pleased to share that we conducted several additional experiments to address these concerns.

Due to the character limit, we have divided our response into two comments:

  • Part 1: Revised computational benchmarks, hardware specifications, and Ego4D timing discrepancy.
  • Part 2: Stronger factor-native baselines and the advantages of unrestricted convolution kernels.

We hope that these additional experiments and clarifications adequately address the reviewer's concerns and provide a stronger basis for reassessing the paper.

Weakness 1: Mismatch between Table 2 and theoretical costs

Prompted by the reviewer's observation, we discovered that the original experiments in Table 2 used a JAX convolution implementation that was not well optimized for CPU execution.

Accordingly, we reran the benchmarks in Table 2 and compared our CP-Conv implementation against optimized PyTorch convolution, cuDNN, and FFT-based alternatives.

For the configuration highlighted by the reviewer (d=3, k=7, R=100, N=128, m=16), the revised CPU benchmark gives approximately 2.39× speedup for CP-Conv and 27.65× for CCP-Conv, substantially below the originally reported 48× and 607×.

We note that the theoretical complexity analysis characterizes arithmetic scaling but does not directly predict wall-clock performance, which also depends on the implementation and hardware. The revised measurements are now more consistent with the theoretical analysis, while still demonstrating the computational benefits of our approach.

The advantage remains substantial in settings where dense intermediate tensors become expensive. For example, in a four-dimensional GPU benchmark, CCP-Conv requires 1.48 ms, compared with 60.45 ms for an FFT-based convolution followed by projection (approximately 40.8× faster).

We have updated Table 2 accordingly and can provide the full benchmark results upon request.

Q1: Hardware specifications

The revised Tables 3 and 13 experiments were conducted on an Intel Core Ultra 9 275HX processor (24 cores, 24 logical processors), using PyTorch with optimized convolution implementations.

We will include the hardware specifications and benchmarking methodology in the revised paper.

Q2: Ego4D timing discrepancy

Following the reviewer's observation, we investigated the discrepancy and found that the original Ego4D and UCF101 experiments had been executed in different environments (local PC versus cluster).

Accordingly, we repeated the supervised classification benchmarks using a consistent PyTorch implementation and five seeds on the same CPU.

For UCF101-10, CCP-Conv and dense convolution require 6.64 s and 155.86 s, respectively. For low-resolution Ego4D, they require 8.98 s and 243.60 s.

Consequently, the dense Ego4D/UCF101-10 runtime ratio is now approximately 1.56×, rather than the originally reported 15.3×.

Both settings use 500 optimization steps, and Ego4D has a 1.5× larger temporal dimension.

The revised measurements resolve the previously unexplained timing discrepancy. We will replace the earlier numbers and clarify that implementation and execution conditions can materially affect practical speedups.

AuthorsOpenReview ↗

Continuation of our response to Reviewer QLfm (Part 2/2)

Weakness 2: Stronger baselines for native CP data

We agree that demonstrating computational feasibility alone is insufficient to establish the usefulness of CCP-Conv on native CP data.

Accordingly, we expanded the Fokker–Planck experiments with stronger factor-native baselines, including linear and RBF Ridge classifiers, log-variance Ridge features, and MLPs trained directly on the CP factors.

Furthermore, we expanded the datasets to provide sufficient training samples and allow the convolutional models to achieve higher accuracy. The FP-05 dataset now contains 20,000 samples instead of 1,000. We also introduced two additional datasets designed to test more challenging dependencies:

  • FP-HO: A dataset involving high-order interactions.
  • FP-Product: A dataset involving nonlinear products of underlying parameters.

We removed FP-035 from the revised comparison, as it exhibited similar complexity to FP-05 when sufficient training data were available, and replaced it with the more challenging FP-HO and FP-Product datasets.

The five-seed test results (mean ± standard deviation, in %) are presented below:

MethodFP-05FP-HOFP-Product
CCP-Conv93.47 ± 3.45100.00 ± 0.0193.80 ± 1.28
Factor MLP99.83 ± 0.0985.17 ± 1.3462.63 ± 0.75
Linear Ridge71.30 ± 0.9175.12 ± 0.3633.92 ± 0.72
RBF Ridge70.86 ± 1.2379.27 ± 0.4747.10 ± 0.58
Log-variance Ridge99.89 ± 0.0433.30 ± 0.8637.51 ± 0.73

CCP-Conv outperforms the evaluated baselines when the relationships become more complex, as demonstrated on FP-HO and FP-Product.

However, on the simpler FP-05 problem, statistical features can outperform CCP-Conv. We acknowledge this limitation and will discuss it explicitly in the revised paper.

Weakness 2 (continued): Advantage of unrestricted kernels over separable kernels

We agree that this is an important point and that the ability to employ unrestricted kernels should be supported by an empirical comparison.

We therefore added an experiment using the pretrained R3D-18 convolutional stem as a reference operator. We fitted separable kernels of different ranks to reproduce its responses on real UCF101 video patches.

Across five seeds, the relative output errors were:

KernelRelative output error (%)
Rank-1 separable26.15 ± 0.06
Rank-4 separable1.96 ± 0.07
Rank-16 separable0.010 ± 0.003

These results quantify the expressivity limitations introduced by low-rank kernel constraints when approximating a pretrained unrestricted convolutional operator.

In an additional controlled frozen-backbone classification experiment on a UCF101 subset, an unrestricted stem achieves 71.49% accuracy, compared with 67.32% for a rank-1 separable stem.

These experiments demonstrate a setting in which preserving an unrestricted kernel provides a measurable advantage. We will include these results in the revised paper, as well as the code for reproducibility.

We thank the reviewer again for these observations, which helped us identify weaknesses in the original benchmarking methodology and substantially strengthen the experimental evaluation.

Thank you for the detailed response and the additional experiments. My point about speed in Weakness 1 was a matter of the reliability of the reported results, and since you corrected the measured values and explained the difference in the Ego4D training time, this concern has been resolved.

This is my personal view, but I would like to evaluate this work from the perspective of the idea of the method and its uses, that is, its potential, rather than its immediate speed. Since this method is still at an early stage, I do not think that hardware-level optimization or extreme speed is necessary. Therefore, I do not think that the reduced magnitude of the speedup in the corrected results lowers the value of the paper. Also, I think that very high-dimensional data, even if it is difficult to use in experiments right now, will be dealt with more and more in the future, and the fact that convolution with an unconstrained kernel can be applied to such data that cannot be handled in dense form is a clear strength. It would be good if the main text also described this point, rather than speed, as the core contribution of the paper.

In that case, I think it is important to convincingly show why an unconstrained kernel is necessary. Since this paper takes a different direction from prior works that constrain the kernel, it is necessary to show that there is a setting where the assumption of prior work, that constraining the kernel does not cause a large loss, does not hold. If this point had already been revealed in the paper's existing experiments, there would have been no need to request additional experiments, but at present I think it is insufficient.

The added R3D-18 experiment is an attempt in this direction, but I think it is not sufficient. If the remaining fixed layers were originally pretrained with an unconstrained stem, it seems that this would be a condition favorable to the unconstrained kernel. Also, classification accuracy was compared only with rank-1, which is the most extreme constraint, and the result that the output error becomes small even when the rank is increased only slightly could also be read as supporting the assumption of prior work instead.

Therefore, it would be good to show that, when compared with a constrained kernel at the level actually used in prior work, an unconstrained kernel has a meaningful advantage under conditions that do not favor either side, and the reason for it. Even if it is not an extensive experiment, I think a persuasive example and one or two simple experiments would be sufficient, and it seems there may also be ways such as citing the limitations of kernel constraints reported in prior work as evidence, or showing it without redoing the full training. And it would be good if you explained the strengths that CCP-Conv has in such a setting.

Regarding the first part of Weakness 2, I think it has been resolved by adding baselines to the FP experiments. The following two are points of confirmation and suggestion that do not affect the evaluation. In the added experiments, the response does not state what the inputs of the Factor MLP and the Ridge classifier are; do they take the CP factors directly as input? Also, if the generation method and generation code for the newly added FP-HO and FP-Product datasets were released together in the appendix, it seems it would be helpful for checking and reproducing the results.

I am willing to raise the score even with the current response alone, but since the revised version has not been uploaded yet, I will keep the score for now. Once a revised version reflecting the results reported in the response is uploaded, I will check it and then judge again, together with the response on the necessity of an unconstrained kernel.

Comments on the paper

2 comments

Dear Reviewers and Area Chair,

We would like to inform you that we have uploaded a revised version of our manuscript, together with updated supplementary materials and anonymized reproduction code.

The revision incorporates the additional experiments, analyses, and clarifications discussed in our responses, including:

  • Updated computational benchmarks, with additional PyTorch, cuDNN, and FFT comparisons.
  • Stronger factor-native baselines on the Fokker–Planck datasets, including the additional FP-HO and FP-Product experiments.
  • A controlled PDEBench experiment comparing unrestricted convolution kernels against separable kernels of different ranks.
  • Additional ablations and methodological clarifications, including further details on experimental protocols, computational costs, and limitations.

The updated supplementary code includes the implementations of these additional experiments, along with reproduction instructions.

We sincerely thank all reviewers for their detailed and constructive feedback. Your comments have helped us substantially strengthen the experimental evaluation and clarify the contributions and limitations of our approach.

We hope the revised manuscript and supplementary materials address your concerns, and we would greatly appreciate any further feedback.

Best regards,
The Authors

Dear Area Chair,

We would like to thank all four reviewers for their careful evaluations and constructive engagement throughout the discussion.

In response to their feedback, we substantially revised the manuscript and supplementary materials, addressing the principal concerns through:

  • Corrected computational benchmarks, including optimized PyTorch, FFT, and cuDNN comparisons.
  • A stride-matched R3D-18 experiment, with explicit training-time and preprocessing accounting.
  • Stronger factor-native baselines and additional high-dimensional Fokker–Planck experiments.
  • Controlled comparisons between unrestricted and separable convolution kernels across several ranks, including an axis-aligned control.
  • Additional evidence from real photographic observations, together with expanded discussions of related work, reproducibility, and limitations.

We also clarified the scope of our computational claims. In particular, the reported R3D-18 training-time reduction characterizes the measured CPU training pipeline rather than isolated operator speedup or end-to-end savings including CP decomposition.

Our central contribution is not a new multilinear identity, but the formulation and implementation of trainable, unrestricted-kernel convolution directly on CP representations, with fused compression that avoids materializing large dense inputs and convolutional outputs.

We are grateful that the reviewers engaged with these additions and reconsidered their assessments. We hope the revised manuscript provides a clearer and stronger basis for evaluating the contributions, empirical evidence, and limitations of our work.

Thank you for your consideration.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.