acceptodds
Under review as a conference paper at ICLR 2027

Sharp Scalar Reconstruction Rates and Fusion Separation under Lipschitz Constraints

Abstract

In multimodal prediction, observations may be compressed separately before fusion or jointly after fusion. We ask whether the Euclidean dimension of a nonlinear representation alone imposes a nontrivial information constraint. Without a fixed Lipschitz or network-complexity budget, scalar neural coding extends to arbitrary finite-dimensional observation laws and square-integrable responses. The joint risk infimum equals the full-observation Bayes risk, while separate encoding reduces the dimension budget to modality selection. Under a Lipschitz product budget, however, the approximation problem becomes quantitative. For a uniform input on with , the optimal mean squared reconstruction error through a scalar bottleneck is of order when the product of the encoder and decoder Lipschitz constants is at most . The lower bound holds for all Lipschitz maps, and finite ReLU networks achieve the same rate. For two independent uniform three-dimensional modalities and an additive response, separate encoding with total latent dimension two has risk of order , whereas joint encoding under the same budgets attains . Linear Gaussian frontiers, exact continuous factorization dimensions, and matching finite-bit Gaussian risk exponents provide complementary characterizations of representation cost. Seven controlled experiments examine the constructions and Gaussian benchmarks, showing a decreasing attained joint-to-separate risk ratio under matched Lipschitz budgets and finite-bit decay rates close to the rank-based predictions.

Primary area:
learning theory
Keywords:
Representation Learning, Multimodal Fusion, Lipschitz Constraints

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_K2sHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies how joint versus separate compression of multimodal observations affects prediction accuracy. Its central argument is that limiting the dimension of a representation alone does not guarantee meaningful compression if encoder and decoder complexity and precision are unrestricted. Additional constraints, such as Lipschitz regularity or finite communication rates, lead to more meaningful trade-offs.

Contributions:

1.The authors show that even a one-dimensional representation can approach the optimal prediction risk when model complexity and precision are unrestricted. For separate encoding, they characterize which modalities benefit from receiving nonzero-dimensional representations.

  1. With Lipschitz-constrained reconstruction and for uniform data in dd dimensions, the paper derives the optimal scaling of scalar reconstruction error with the encoder--decoder Lipschitz budget, supported by matching lower and upper bounds.

  2. A two-modality example shows that joint encoding can achieve a better asymptotic prediction rate than separate encoding, although the optimal joint rate remains unknown.

  3. The paper analyzes prediction risk under Gaussian models and finite communication budgets, showing how performance depends on the structure of the prediction task.

  4. Several theoretical examples and controlled experiments illustrate the predicted behavior under different resource constraints.

Critical strengths
  1. The paper clearly distinguishes representation dimension from actual compression, emphasizing the need to constrain precision, complexity, or communication rate.

  2. The matching bounds for scalar reconstruction under a Lipschitz constraint provide a precise characterization of how reconstruction quality scales with the available resources.

  3. The example demonstrates that joint encoding can outperform separate encoding under the same overall resource constraint.

  4. The paper connects nonlinear representations, Lipschitz constraints, and finite-bit compression within a common prediction framework.

Critical weaknesses
  1. Very difficult to read. The paper needs better presentation, especially for those who are not familiar with this line of research.

  2. The paper builds on existing scalar-coding results. It should more clearly explain which contributions go beyond prior work, particularly regarding general data distributions and modality selection.

  3. The lower bound for scalar reconstruction is central to the paper. The authors should clarify the treatment of degenerate encoders, atoms in the encoded distribution, and the geometric assumptions behind the bound.

  4. The paper establishes an advantage for joint encoding but does not prove that its achievable rate is optimal. A matching lower bound would strengthen the result.

  5. A Lipschitz constraint does not directly limit network size or parameter magnitudes. The paper would benefit from analyzing how model complexity grows as prediction error decreases.

  6. Broader comparisons and tests across different distributions and prediction tasks would provide stronger evidence of the results' practical relevance.

  7. The authors should explain more clearly how prediction-task structure, covariance conditioning, and bit allocation affect the reported rates.

Questions for authors

See all the weaknesses above and additionally,

  1. How do the dimension-only results differ from existing scalar-coding methods?
  2. How does the required network size scale with the Lipschitz budget and target error?
  3. What assumptions are essential for the finite-bit Gaussian rate results?
Reviewer_cXENOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors compare joint and separate representations in multimodal prediction under shared latent-dimension and regularity constraints. The paper first establishes that, without bounds on network complexity or Lipschitz constants, a one-dimensional neural bottleneck can approach the full-observation Bayes risk, while separate encoding reduces the representation budget to a modality-selection problem. The principal theoretical contribution is a sharp characterization of scalar reconstruction under a Lipschitz product budget. For uniform inputs on a multidimensional unit cube, the authors prove matching lower and constructive finite-ReLU upper bounds: Ed(Λ)≍Λ−2/(d+1).E_d(\Lambda)\asymp\Lambda^{-2/(d+1)}. They use this result to construct a quantitative fusion separation: for two independent three-dimensional modalities and an additive target, separate encoding has risk of order Λ−1/2\Lambda^{-1/2}, whereas joint encoding attains an upper bound of order Λ−2/3\Lambda^{-2/3} under the same budgets. Complementary results examine continuous factorization, Gaussian linear representations, and finite-bit coding. Controlled experiments illustrate the proposed rates and fusion trade-offs.

Critical strengths
  1. The sharp scalar reconstruction theorem is a good mathematical contribution. Its matching lower bound for all Lipschitz encoder-decoder pairs and constructive finite-ReLU upper bound provide a precise characterization rather than a one-sided approximation estimate.

  2. The joint-versus-separate fusion separation translates the reconstruction rate into a clear architectural implication. The distinction between dimension-only expressivity and Lipschitz-constrained approximation is conceptually insightful.

Critical weaknesses
  1. The strongest nonlinear rate theorem is restricted to scalar bottlenecks and uniform distributions on a unit cube.

  2. The fusion example is deliberately structured, and the results are primarily population approximation statements.

Questions for authors

Is there an expected sharp reconstruction exponent for general bottleneck dimensions 1<r<d1<r<d, and can the analysis be extended beyond uniform input distributions?

Reviewer_sSLeOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper compares joint and separate encoders for multimodal prediction when the total bottleneck dimension is limited. My understanding is that, without additional constraints, even a single real-valued coordinate can approach the predictive performance of observing all inputs. This motivates introducing a budget on the product of the encoder and decoder Lipschitz constants. The main result gives the sharp mean-squared reconstruction rate Λ^{-2/(d+1)} for uniform data on a d-dimensional cube through a scalar bottleneck. The authors then use this result to construct an example in which joint encoding achieves a better rate than separate encoding. The paper also considers linear, Gaussian, continuous-factorization and finite-bit settings, with experiments that mostly evaluate explicit constructions.

I had difficulty following parts of the setup, so this summary only reflects my current understanding. I'll put more details in the following review. Overall the bad writing made it difficult to see how all the results fit together and how broadly the conclusions about multimodal fusion should apply. As I understand it, the fusion separation is established for one specific example, and the optimal rate for the joint encoder remains open.

Critical strengths

Theorem 5 gives matching upper and lower bounds. The rate Λ−2/(d+1)\Lambda^{-2/(d+1)} is sharp: the lower bound holds for all Lipschitz encoder/decoder pairs (not just networks), and an explicit ReLU construction attains it.

Critical weaknesses
  1. Basic setup is unclear. Several key terms are never properly defined:
    • Data model: how does YY depend on XX? Section 3 only assumes a general joint law, and later sections quietly switch settings (Y=XY = X in Thm 5, Y=(X1+X2)/2Y = (X_1+X_2)/\sqrt{2} in Thm 6, Gaussian in Sec. 6–7).
    • NNa,b\mathrm{NN}_{a,b}: described in one sentence, with no formal definition (layers, parameters, skip connections?).
    • Bottleneck dimension: only implicit from the signatures ϕ∈NNp,r\phi \in \mathrm{NN}_{p,r} and ϕm∈NNpm,rm\phi_m \in \mathrm{NN}_{p_m,r_m}, yet it is the central quantity of the paper.
  2. I am not sure how general the fusion result is. Theorem 6 considers two independent uniform inputs, an additive response, and r=2r = 2. I understand that the bounds show an advantage for joint encoding in this case, but which parts of this setup are essential? It is hard for me to tell when to expect a similar advantage in other settings. The paper also leaves the optimal joint rate open, so I am unsure how large the gap could actually be. More generally, I had trouble identifying what is new in the other results, since several are described as extensions of earlier work or standard calculations. It would help to explain what these results add and how they support the main argument.
  3. I am unsure what to take away from the experiments. Most experiments evaluate constructions supplied by the theory, so I read them mainly as checks of those constructions. Experiment 4 seems more relevant to what happens when networks are trained, but the scalar-bottleneck network has a much higher excess risk than the construction (0.189±0.0300.189 \pm 0.030 versus 0.003). I could not tell whether this gap comes mainly from optimization, the parameter bounds, or the network architecture. What should we conclude from it about the usefulness of the scalar bottleneck in practice? It would also help to test the predicted Lipschitz rate with trained networks whose Lipschitz constants are controlled.
  4. The settings behind the main results are very specific. Theorem 5 studies reconstruction of uniform data on a cube through a scalar bottleneck, while Theorem 6 uses independent uniform modalities and an additive response. I am not sure which assumptions are needed for the proofs and which are essential to the conclusions. For example, how much should we expect to carry over to nonuniform data or a different relationship between the inputs and the response? I would also appreciate more explanation of the Lipschitz-product budget: how should I interpret it for a trained network, and why is it the relevant constraint here? Since the theory concerns population risk, I am also unclear about what it implies when the model is fitted from finite data.
Questions for authors
  1. What is the data model? When (X,Y)(X, Y) is introduced in Section 3, please say at least briefly how YY is assumed to depend on XX, and state in each section which specific model the results assume (e.g., Y=XY = X in Thm 5, Y=(X1+X2)/2Y = (X_1+X_2)/\sqrt{2} in Thm 6, jointly Gaussian in Sec. 6–7).
  2. What exactly is the model class NNa,b\mathrm{NN}_{a,b}? Please give a formal definition (layer-wise composition, parameters, whether skip connections are allowed), define the bottleneck dimension rr (and rmr_m) explicitly, and state formally how the Lipschitz-budgeted classes of Section 5 are built from it.
  3. What is the intended message of Theorem 2? Is it that a scalar code already preserves the full predictive capacity, so extra bottleneck dimensions don't help (beyond choosing which modalities to keep)? If so, please state this takeaway explicitly.
  4. Can training actually reach the Theorem 2 infimum? In Exp. 4 the trained scalar-bottleneck networks stay far from it. Is there any evidence (longer training, larger networks, Lipschitz regularization) that it is reachable, or should readers conclude it is not achievable in practice?
  5. Finite precision: a scalar code carries unlimited information only with infinite precision, and Thm 12 handles bits only for the Gaussian model. Is there a bit-precision version of Lemma 1 / Thm 2 for general laws, i.e., how many bits does the scalar code need for excess risk ε\varepsilon? And what happens when a bit budget and a Lipschitz budget are imposed together?
  6. Why is the Lipschitz product Lip(ϕ) Lip(ψ)\mathrm{Lip}(\phi)\,\mathrm{Lip}(\psi) the right complexity measure for neural encoders/decoders? Do the conclusions survive under other measures, e.g., network size or weight norms? Relatedly, do Thms 5–6 extend beyond the uniform cube, and beyond independent modalities with an additive response?
  7. Is the joint rate Λ−2/3\Lambda^{-2/3} in Theorem 6 optimal? Without a lower bound for the joint architecture, how large is the actual gap between joint and separate encoding?
  8. Novelty of Thm 5 vs. Lipschitz widths (Petrova & Wojtaszczyk 2021) and stable nonlinear widths (Cohen, DeVore, Petrova & Wojtaszczyk 2022): those works already study Lipschitz encoders/decoders. What exactly is new here beyond moving from worst-case to average-case error with a fixed scalar latent?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.