acceptodds
Under review as a conference paper at ICLR 2027

Sharp Rates for Information Composition under Source Uncertainty

Abstract

Suppose each view of an observation is compressed on its own before the views are combined for prediction. Even when each compressed view loses very little task information, the combined representation can lose much more. We study when that can happen for a fixed finite source and deterministic encoders. For finite positive failure depth d, the sharp relation between local and joint information loss has exponent 1 - 2^(-d). Infinite depth gives a linear relation. At depth zero, zero local loss can still come with positive joint loss. For rational inputs, the depth can be computed in polynomial time, and two views are enough to realize every finite depth. We then allow the source distribution to move within total variation e. For a depth-two eight-atom source, with local-loss budget u and relative posterior floor tau, the sharp joint rate is minu^(3/4), tau^(-1/3)u + e log(1/e). Small changes in source mass can change the exact depth without changing the support. Under log loss, the same exponent sets the sample-size scale where raw and compressed decoders trade places, while the crossing point depends on the smoothing rule. Existing semialgebraic and conic results cover part of the general machinery. The result here gives the exact exponent from the source/encoder structure and quantitative bounds when the source is uncertain.

Primary area:
learning theory
Keywords:
information theory, representation learning, deterministic compression, information loss, distributional robustness

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_tny3OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

As far as I can tell the paper seems to study if several observations are compressed separately and then the compressed representations are used for a prediction task, under what conditions do zero local losses imply zero joint loss, and how large the joint loss can be when the local losses are small.

Critical strengths

None.

Critical weaknesses
  • I have significant concerns about this paper and its integrity. Beyond the first few paragraphs'' (perhaps beyond the first paragraph of the intro), the writing doesn't make sense.

  • The formal problem formulation is not clear and introduces a lot of terms that are not described explicitly. There is a lot of technical jargon that is not defined explicitly. For example, in the first paragraph of Section 3, it is mentioned that "The raw tuple distinguishes every atom." However, I am not clear what "atom" means in this context.

  • Similarly, in the same paragraph it is stated that "let J contain the nonempty fibers of Z". However, "J" is not defined yet. Also, what is a "nonempty fiber"? The rest of the paper goes on like this. For example equation (1) describes two mutual information quantities that depend on some term "\mu" but we do not know what "\mu" is. Perhaps it is related to another term "\mu_x" that is used in the first paragraph, but then we are also not told what "\mu_x" is, or how "\mu_x" relates to "\mu". Then the paper moves on to talking about a "target map" B_T without explicitly defining what is the target.

  • The entirety of the paper goes on like this. Beyond the first paragraph of the introduction, I cannot confirm what the paper is trying to do. In fact, finally when I reached to the AI statement I saw the sentence "Human verification and author approval remain to be completed.", which is quite concerning. The authors do not seem to value the reviewers' and AC time.

Questions for authors

I am not sure what can be done to save this paper. I am not sure why there is a sentence "Human verification and author approval remain to be completed." in the AI statement. I would like to believe this was not intentional, but if authors read the paper they would have caught it. So the most logical explanation is that this paper was submitted without author verification.

I think the best suggestion is please do not submit AI assisted papers without author verification.

Reviewer_b1iTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper theoretically studies whether preserving task information in separate compressed observations also preserves information when those observations are combined under a fixed finite source and deterministic encoders. The main result addresses the relationship between the local information loss and the joint loss. The posterior floors, uncertainty in the source distribution, and the consequences for prediction are also investigated.

Critical strengths

The paper addresses an interesting problem: If each observation is compressed with little loss of predictive information, how much information can be lost when the compressed observations are used together under a fixed finite source and deterministic encoders?

The paper derives the exact exponent of the relationship between the local information loss and the joint loss by failure depth. In addition, the polynomial-time depth computation makes the structural result more useful.

Critical weaknesses

The paper presentation is difficult to follow for readers outside of this field and may not be interesting for the machine learning audience.

The paper should benefit from an early example illustrating local versus joint information loss, followed by a step-by-step explanation of how failure depth determines the exponent. It would improve the accessibility of the paper.

The manuscript is dense and needs a clearer proof roadmap. For instance, several important arguments appear later in the appendix.

Questions for authors

N/A

Reviewer_SGDDOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies information composition for multi‑view deterministic encoders. The authors introduce the notion of failure depth for a finite source‑encoder system, prove for finite positive depth dd, the sharp exponent is α=1−2−d\alpha=1-2^{-d}; infinite depth yields linear bound α=1\alpha=1, while depth‑zero systems exhibit complete composition failure. The depth can be computed in polynomial‑time using rational linear programming. The paper further proves that for a depth-two eight-atom source, with local-loss budget uu and relative posterior floor tau, the sharp joint rate is minu(3/4),tau(−1/3)u+elog(1/e)min{u^(3/4), tau^(-1/3)u} + e log(1/e). where ee is the total variation of the source distribution.

Critical strengths

The definition of failure depth provides a clean structural invariant to characterize information composition. The sharp exponent classification theorem is a substantial conceptual contribution, clarifying when local loss controls joint loss and what power‑law rate to expect. The distinction between finite depth, infinite depth and depth‑zero failure is well‑motivated and mathematically rigorous.

Critical weaknesses

All theory is built for finite discrete sources and deterministic encoders. The paper does not extend results to continuous sources, stochastic encoders, or learned neural multi‑view representations. It remains unclear how to approximate or estimate failure depth on real‑world datasets, given the proven instability under source perturbations. While this is fundamental theoretical work, the discussion about bridging to practical machine learning could be expanded.

Questions for authors

The exact failure depth cannot be consistently identified from finite samples even with fixed support. Are there computationally feasible approximate surrogates for depth that could be estimated from data, with provable guarantees?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.