acceptodds
Under review as a conference paper at ICLR 2027

The Hidden Perception Constraint in Task-Aware Compression

Abstract

With the recent advancements of neural compressors, explicitly incorporating perception constraints into the design of compression schemes has gained significant attention. Traditionally, these perception constraints ensure that the distribution of the reconstruction does not significantly deviate from the distribution of the source thus attesting to the perceptual quality of the reconstruction. In this work, we uncover several perception constraints that are naturally present in task-aware compression. In particular, we consider a problem where the primary task is reconstruction and the secondary task is classification (or a statistical test). We study this problem at varying levels of domain information available to us and discuss how to utilize the naturally emerging perception constraints to design rate minimal compression schemes that also maximize the utility of our secondary task. We show that in this setting, if the decision boundaries of the classifier are ill-defined (mismatch) for our source distribution, then matching onto a target distribution enhances our classification accuracy.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Cross-domain, perception, compression

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_m8vkOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper systematically investigates the task-aware compression problem with the reconstruction and classification constraints (i.e., a statistical test). The paper shows the coding theorem for the rate-distortion-error function. The proposed problem is also studied at various levels of available domain information (individual source hypotheses and the classifier's decision regions at the receiver side). An interesting result is that minimizing the distance between the target distribution of the statistical test and the source distribution enhances classification accuracy, with an explicit bound derived. The theoretical claims are supported by numerical experiments

Critical strengths

The quality of the paper is good. The paper presents a clear and well-structured theoretical analysis by showing a new coding theorem of the rate-distortion-error function and deriving the closed-form function for the Bernoulli case.

The probability of error bounds for a known test distribution with the distribution matching to improve the classification accuracy are interesting and insightful. The proposed problem is also studied at various levels of available domain information, including individual source hypotheses and the classifier's decision regions at the receiver side.

The numerical results and empirical validation are convincing. The paper is well written, with a clear and well-organized presentation.

Critical weaknesses

In the related work section, the comparison with the perception constraint in previous works (Liu et al., 2022; Nguyen et al., 2026) is a mismatch. This paper minimizes the distance between reconstruction probability and the target probability, while the previous paper minimizes the distance between reconstruction probability and the source probability, aiming to improve the perception quality.

The role of the shared common randomness is not explained clearly in the framework in the main paper. It helps show the coding theorem by applying the Strong Functional Representation Lemma.

The mathematical derivations are rigorous, though additional intuition would make them more accessible to the machine learning audience.

Questions for authors

The paper should benefit from studying the closed-form solution of the rate-distortion-error function for a Gaussian source.

The proposed problem is in the one-shot setting. Could the authors give some insight about the asymptotic setting? It should be a good extension.

I am wondering whether more results for the rate-distortion-error problems in (10) and (11) can be derived, such as the closed-form solutions for Bernoulli and Gaussian sources.

3 comments
AuthorsOpenReview ↗

Reply to Reviewer m8vk

We thank the reviewer for the very positive assessment and for suggestions that improve the presentation. All changes described below will appear in the revised PDF, which we will upload by November 15.

W1: Related work. We agree that the paragraph was unclear and have rewritten it. Most rate–distortion–perception works we cite (e.g. Theis & Wagner, 2021; Wagner; Hamdi et al., 2026) match the reconstruction to the exact source distribution, as the reviewer states, to improve the perceptual quality of the reconstruction. The works of Liu et al. (2022) and Nguyen et al. (2026) are slightly different. While they enforce the perception constraint to improve the perceptual quality, the distribution to which they match is not exactly the source distribution (in Liu et al., they match to a clean distribution of which the source is a degraded version). We share the target-matching aspect with them. The difference is where the target comes from and why it is matched: in our setting the target is class-conditional, set by the distribution the receiver's classifier was trained on, and the matching constraint follows from the classification constraint rather than being imposed for perceptual quality. The revised paragraph separates the two lines of work in this way.

W2: Common randomness. We added a paragraph after Def. 1. For R(D,α)R(D, \alpha) alone, deterministic coding schemes would be sufficient and would not require common randomness. We use common randomness to simplify our achievability proofs, and these same coding schemes can be used when perception constraints are involved, as in (7), (10) and (11). We have now outlined the main steps of the coding scheme in the appendix.

Intuition. We added two short explanations for readers outside information theory. After (5), we explain the two parts of the optimal rate:

R(D)⏟smallest rate thatachieves pe∗=H(J)⏟conveys the transmitter’sown decision J+R′(D)⏟reconstructs the sourcewithin each decision region\underset{\text{smallest rate that} \atop \text{achieves } p_e^{\ast}}{\underbrace{R(D)}} = \underset{\text{conveys the transmitter's} \atop \text{own decision } J}{\underbrace{H(J)}} + \underset{\text{reconstructs the source} \atop \text{within each decision region}}{\underbrace{R'(D)}}

To reach the best possible error, a code must at least convey the transmitter's own decision JJ, which costs H(J)H(J) bits; the remaining rate R′(D)R'(D), defined in (6), is spent on reconstructing the source within each decision region. After Thm. 1, we explain the three terms of the error bound:

pe(X^)≤pe∗⏟transmitter’sown error+Cre⏟receiver’s erroron its own domain+∑i=0M−1Pi∥PX^i−ri∥TV⏟how far the reconstructions of class iare from class i in the receiver’s domainp_e(\hat X) \le \underset{\text{transmitter's} \atop \text{own error}}{\underbrace{p_e^{\ast}}} + \underset{\text{receiver's error} \atop \text{on its own domain}}{\underbrace{C r_e}} + \underset{\text{how far the reconstructions of class } i \atop \text{are from class } i \text{ in the receiver's domain}}{\underbrace{\sum_{i=0}^{M-1} P_i \Vert P_{\hat X_i} - r_i \Vert_{TV}}}

That is, the error splits into three parts, namely the transmitter's own error, the receiver's error on its own domain, and how far each class's reconstructions are from that class in the receiver's domain. Only the last depends on the codec, which is why matching onto rir_i lowers the error.

Q1: Gaussian sources. This is an interesting suggestion that would shed some light on the behavior of the RDE formulation in the continuous domain. We hope to explore this in the future and to compare the solution of the RDE problem formally with that of our RDP formulation in (10).

Q2: the asymptotic setting. The RDE formulation is derived from the achievability definition given by (1), which is in the asymptotic setting. In Proposition 1, using the Strong Functional Representation Lemma, we show that the rate defined by (2) is the asymptotic rate of the problem.

Q3: closed forms for (10) and (11). Solving (10) and (11) with Gaussian priors is a useful and interesting extension. However, this would also require solving an RDP problem for Gaussian mixtures, which might be challenging. We have added this interesting extension as future work. The formulation with Bernoulli priors is quite possible, but it may result in degenerate formulations or trivial solutions since ∣X∣=2|\mathcal{X}|=2.

We thank the reviewer again for the support and the suggestions.

I thank the authors for their detailed clarifications, which I found convincing, particularly their explanation of the formula’s intuition. I was mistaken about Question 2 in the asymptotic setting, and I appreciate the authors’ correction. After reviewing their responses to the other reviewers as well, I will maintain my positive evaluation and current score.

We sincerely thank reviewer m8vk for their helpful suggestions and their positive assessment of our work. We greatly appreciate your time and effort put into reviewing our paper.

Reviewer_qSZtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies compression when the reconstruction must remain close to the source while also working well with a fixed, mismatched classifier. It considers three levels of access to source distributions and receiver information.

The main contribution is a connection between classification requirements and distribution-matching constraints. The authors characterize a rate–distortion–error function, derive an error bound using class-conditional distribution matching, and examine a weaker mixture-matching setting. Validation includes discrete examples and SVHN-to-MNIST compression experiments.

Critical strengths

Adapting compressed reconstructions to a fixed third-party classifier is a sensible setting, particularly when receiver access is limited.

Theorem 1 explicitly relates classification error to source classification error, target-domain error, and class-conditional distribution mismatch.

The neural experiments do not use its gradients, which respects an important restriction of the proposed setting.

Critical weaknesses

The “equivalent RDP” framing is too broad. Finite KL divergence in Equation (7) enforces support restrictions, but also imposes additional distributional constraints. Equation (10) is a sufficient construction based on an upper bound, as the authors acknowledge. Equation (11) has no corresponding classification guarantee. These are different relationships, rather than a general equivalence between classification and perception constraints.

The neural experiments do not directly validate the theoretical guarantee. Theorem 1 uses total variation, and Corollary 1 uses KL divergence. The experiments instead optimize an adversarial Wasserstein-1 estimate. Small Wasserstein distance alone does not control classifier error without further assumptions on the decision boundary or margins. The results illustrate the idea, but do not establish the stated bound in this setting.

The source classifier is treated as Bayes optimal without justification. Section 7 identifies its observed error, 0.079, with (p_e^*). A trained ResNet is not known to attain Bayes error. Its error can serve as a practical reference, but should not be presented as the theoretical optimum.

The Level 3 information restriction is unclear experimentally. Level 3 assumes access only to mixture distributions, yet the distortion measure uses features from a classifier trained on labeled SVHN data. This supplies class information indirectly. The paper needs to explain whether such a pretrained feature extractor is allowed and what remains possible without it.

Experimental reporting needs reconciliation and stronger controls. Table 4 lists rates of 24, 32, 80, and 192 bits, whereas the main figures use 24, 64, 96, and 128 bits. Also, (J) is a predicted label and need not be uniformly distributed, even with uniform true-class priors, so (H(J)=\log_2 M) is not automatic. A label-conditioned reconstruction baseline would help establish how much the proposed matching objective adds when (J) is transmitted.

Questions for authors

Can you state the assumptions needed for Lemma 1, including posterior ties and target-support coverage?

Which results are exact equivalences, which are sufficient conditions, and which are heuristics?

Can the Wasserstein experiments be connected to the error bound through a margin assumption or an additional diagnostic?

What labeled information is permitted in Level 3? Does the result survive with an unlabeled or fixed generic distortion representation?

Can you reconcile the reported rates and compare against a simple decoder conditioned on the transmitted predicted label?

2 comments
AuthorsOpenReview ↗

Reply to Reviewer qSZt (Part 1)

We thank the reviewer for a detailed and precise review and for the important questions raised. We have revised our paper accordingly and answer the reviewer's concerns and questions below. All changes described below will appear in the revised PDF, which we will upload by November 15.

W1, Q1 and Q2: equivalences, sufficient conditions and heuristics. We agree that the term "equivalent RDP" was too broad. As correctly noted by the reviewer, the equivalence only holds under certain conditions. We have rephrased the statements that indicated an exact equivalence between the RDP and RDE formulations. We now state precisely when equivalence holds, when a formulation is a sufficient surrogate, and when it is simply a heuristic.

  • Under the zero posterior-ties (ZPT) assumption, which we state below, Lemma 1 and (5) are exact characterizations.
  • Under the ZPT assumption, (7) recovers the RDME problem as P→∞P \to \infty in the discrete domain (q(x)>0 ∀x∈Xq(x)>0 \, \forall x\in \mathcal{X} is assumed). In the continuous domain, it only forms a sufficient surrogate that achieves the optimal error.
  • Thm. 1 and Cor. 1: These are rigorous upper bounds on pep_e that do not rely on the ZPT assumption.
  • Since R(D,α)≤R(2)(D,α)R(D, \alpha) \le R^{(2)}(D, \alpha), the RDP formulation in (10) is a sufficient surrogate for the RDE formulation in (2).
  • The RDP formulation in (11) is only a heuristic with no error guarantee. The KL divergence in the perception constraint may be replaced by another perception measure where appropriate.
  • The neural objective is a practical surrogate and can now be linked to our theory with the newly added Corollary 2 (see W2 and Q3 below).

Q1: Assumptions for Lemma 1 and (7).

  • Posterior ties. Lemma 1 was derived under the assumption that P(πipi(X)=πjpj(X))=0P(\pi_i p_i(X) = \pi_j p_j(X))=0 for i≠ji \neq j (ZPT assumption), which is why, in the proof of Lemma 1, we took the function lj,i=πipi(x)−πjpj(x)l_{j,i}=\pi_i p_i(x) - \pi_j p_j(x) to be strictly negative in region Rj∗R_j^*. This assumption was missing from our original text, and we thank the reviewer for raising this important concern. Under this assumption, Lemma 1 and (5) hold as stated; all our examples satisfy it. Without this assumption, the condition in Lemma 1 is sufficient to achieve the Bayes-optimal error, but not necessary, since a point with a posterior tie may be mapped into the region of any of its tied classes without raising the error. Hence, the ZPT assumption is necessary for the equivalence in (7) to hold.
  • Finite KL and Target-support coverage. As correctly noted by the reviewer, a finite KL may enforce other constraints in addition to (4). For example, in the continuous domain, it requires PX^i≪q~iP_{\hat X_i} \ll \tilde q_i, which implies (4) but also excludes some stochastic mappings that satisfy (4). Therefore, in the continuous domain, even with the ZPT assumption, (7) would only be a sufficient surrogate of the RDME formulation. However, this problem does not arise if we operate in the discrete domain. In this case, the finite KL constraint covers all possible mappings that achieve the Bayes error when the ZPT assumption holds and when q(x)>0 ∀x∈Xq(x)>0\,\forall x\in \mathcal{X}. Then, (7) recovers the RDME problem as P→∞P\to \infty. We now make this distinction explicit in Section 4.

W2 and Q3: the neural experiments and the guarantee. We agree that a small W1W_1 distance does not necessarily imply a small TV distance in general. However, for finite metric spaces, there exists a useful relationship between the TV and W1W_1 distances (Gibbs & Su, 2002, Thm. 4). Using this relationship, we obtain the following upper bound: pe≤pe∗+Cre+1dmin∑iPiW1(PX^i,ri)p_e\leq p_e^*+Cr_e+\frac{1}{d_{min}}\sum_{i}P_iW_1(P_{\hat X_i}, r_i) where dmind_{min} is the smallest distance between two distinct elements of the space. Since our images take values in a finite set, this new bound can be linked to our neural objective. We have now added this new bound as a corollary (Corollary 2) to Thm. 1.

W3: the source classifier is not Bayes optimal. We completely agree with the reviewer. Since the error of any transmitter-side test gives an upper bound for pe⋆p_e^\star, the upper bound in Thm. 1 would still be valid if we simply replaced pe⋆p_e^\star with the error of the transmitter-side test. We now state this explicitly in Thm. 1, and Sec. 7 and the figures now describe 0.079 as the transmitter-side classifier's error, a practical reference rather than the optimum.

AuthorsOpenReview ↗

Reply to Reviewer qSZt (Part 2)

W4 and Q4: labelled information at Level 3. We agree with the reviewer on the information leakage present within Level 3. Level 3 is defined explicitly to use only unlabeled source samples and samples from the receiver's domain. The distortion in our original Level 3 experiment used features of gXg_X, which was trained with SVHN labels, and the layer was chosen by class separation; as the reviewer says, that is Level 2 information, and Sec. 7 now says so. To see whether the result survives without it, we re-ran Level 3, and Level 2 without JJ for comparison, with three label-free distortions: features of an encoder trained with SimCLR (Chen et al., 2020) on the same SVHN training images without labels, an edge map (Sobel), and pixel MSE. All runs use R=128R = 128 bits, and we report the average pep_e over two seeds (chance level is pe=0.90p_e=0.90):

DistortionLabels usedLevel 3Level 2, no JJ
gXg_X features (paper)yes0.3470.228
SimCLR featuresno0.4420.4420.2460.246
Edge map (Sobel)no0.7960.7960.317
Pixel MSEno0.8000.8000.2970.297
  • With SimCLR features, the result survives. Level 3 reaches pe=0.442p_e=0.442 for SimCLR features, compared with pe=0.347p_e=0.347 using gXg_X and pe=0.90p_e=0.90 without matching, at the same perception (≈7\approx 7). At Level 2 it reaches pe=0.246p_e=0.246, close to the pe=0.228p_e=0.228 obtained with gXg_X.
  • With pixel MSE or Sobel, the result does not carry over to Level 3. Pixel MSE and the edge map stay close to chance (pe≈0.8p_e\approx0.8). With pixel MSE the reconstructions look like MNIST digits, but mostly of the wrong class.
  • At Level 2 the class-conditional distribution carries the class, so even pixel MSE reaches pe=0.297p_e=0.297.

In short, Level 3 needs a distortion that separates the classes, and labels are not needed to get one (App. H.4).

W5 and Q5: rates, H(J)H(J) and a decoder conditioned on JJ.

  • Rates. Figs. 5–7 use (L,K)=(8,8)(L, K) = (8, 8), (16,16)(16, 16), (16,64)(16, 64) and (32,16)(32, 16), that is R′=24R' = 24, 64, 96 and 128 bits, plus H(J)H(J) when JJ is sent; Fig. 10 adds 32, 40, 48, 80 and 160 bits. Table 4 is now replaced by a table containing every configuration trained, with R′R', R′+H(J)R' + H(J), the entropy measured on the codewords (always below the nominal rate, so the plotted rates are upper bounds) and the figures that use it.
  • H(J)H(J). We agree that uniformity is not automatic. On the 3000 SVHN test images, the empirical entropy of the transmitter's predictions is 3.3207 bits, against log⁡210=3.3219\log_2 10 = 3.3219 bits, so charging log⁡2M\log_2 M overstated the cost by 0.0012 bits. We now report the empirical value.
  • A trivial scheme using JJ. We added a comparison of our scheme with several label-only schemes (App. H.3). In each, the transmitter sends only JJ (3.3 bits) and the receiver outputs an MNIST image of class JJ without using any other information about the input: a randomly chosen image of that class, one fixed image chosen for each class, or the average image of the class. A fourth scheme also sends the index of the MNIST image of class JJ that is closest to the input under our distortion (this requires the MNIST dataset to be available at the transmitter as well), which raises the rate to 11.5 bits.
SchemeRate (bits)Distortionpep_eW1W_1
JJ + random MNIST image of class JJ3.31.970.0850.06
JJ + one fixed MNIST image per class3.31.570.07913.1
JJ + mean MNIST image of class JJ3.31.550.53611.6
JJ + closest MNIST image of class JJ11.51.450.0916.0
Ours, send JJ, λ=10\lambda = 1027.31.300.1232.45

These schemes reach the error floor at 3.3 bits, as the reviewer anticipated, but their reconstructions are far from the input (distortion 1.55–1.97). Below a distortion of about 1.41.4, none of them approaches our codec, which reaches pe=0.12p_e= 0.12 at D=1.30D=1.30. Above about 1.451.45, sending a fixed image or the closest MNIST image matches our error at a lower rate, but those outputs are far from the MNIST distribution (W1W_1 between 6 and 13, against 1.0–3.7 for our codecs). The W1W_1 of the random-image scheme is optimistic, because the critic's reference images come from the same pool.

We thank the reviewer again; the questions led directly to the new results above.

References

  1. [Gibbs & Su(2002)] A. L. Gibbs and F. E. Su. Choosing and bounding probability metrics. International Statistical Review, 70(3):419–435, December 2002.
  2. [Chen et al.(2020)] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.
Reviewer_rbg8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work examines a compression system whose receiver applies a pretrained classifier from another domain. Reconstruction quality and classification accuracy must therefore be considered together. The analysis develops different distribution-matching formulations depending on the available domain knowledge, followed by discrete examples and digit-domain experiments.

Critical strengths
  1. The mismatch between the source and receiver gives the compression problem a concrete motivation.
  2. The examples show why matching the target marginal can produce plausible samples while losing the source’s class information.
Critical weaknesses
  1. The paper sometimes presents a sufficient design strategy as a more complete characterization. Equation (10) restricts the problem through an error upper bound, and the manuscript explicitly acknowledges that its minimum rate may exceed the original one. These are useful contributions, but the broad statements about an equivalent perception constraint and optimal compression rates need to preserve those qualifications.

  2. My main concern is the gap between the bound and the neural objective. An adversarial Wasserstein estimate cannot simply replace TV or KL in the classification guarantee. The experiments can demonstrate useful behavior without validating that guarantee, but this distinction should be explicit. Likewise, the transmitter network’s measured error is not enough to establish the Bayes error used in the theory.

  3. The experimental configuration needs reconciliation. The appendix lists rates of 24, 32, 80, and 192 bits, whereas the main plots show 24, 64, 96, and 128 bits.

Questions for authors
  1. What supports the use of Wasserstein training and an empirical transmitter error within the theoretical interpretation?
  2. Can the authors add a label-only reference to the joint rate-distortion-classification evaluation and specify the actual quantizer settings used for each figure?
2 comments
AuthorsOpenReview ↗

Reply to Reviewer rbg8 (Part 1)

We thank the reviewer for the positive assessment and for precise, actionable points. All changes described below will appear in the revised PDF, which we will upload by November 15.

W1: sufficient strategies presented as characterizations. We agree that the exact conditions under which the RDP and RDE formulations are equivalent were not precisely described, and we have reworded the broad statements regarding the equivalent perception constraints and optimal compression rates. We now state precisely when equivalence holds, when a formulation is a sufficient surrogate, and when it is simply a heuristic.

  • We assume that P(πipi(X)=πjpj(X))=0P(\pi_i p_i(X) = \pi_j p_j(X))=0 for i≠ji\neq j and this assumption has been stated as the zero posterior-ties (ZPT) assumption in the revised manuscript.
  • Under the ZPT assumption, (7) recovers the RDME problem as P→∞P \to \infty in the discrete domain (q(x)>0∀x∈Xq(x)>0 \forall x\in \mathcal{X} is assumed). In the continuous domain, it only forms a sufficient surrogate that achieves the optimal error.
  • Thm. 1 and Cor. 1: These are rigorous upper bounds on pep_e that do not rely on the ZPT assumption.
  • Since R(D,α)≤R(2)(D,α)R(D, \alpha) \le R^{(2)}(D, \alpha), the RDP formulation in (10) is a sufficient surrogate for the RDE formulation in (2).
  • The RDP formulation in (11) is only a heuristic with no error guarantee. The KL divergence in the perception constraint may be replaced by another perception measure where appropriate.
  • The neural objective is a practical surrogate and can now be linked to our theory with the newly added Corollary 2 (see W2 and Q1 below).

W2 and Q1: Wasserstein training and the empirical transmitter error.

  • Wasserstein training. We agree that the experiments were somewhat disconnected from the theory due to the use of the W1W_1 distance. We have now added a new corollary (Corollary 2) to Thm. 1, which addresses this issue by exploiting the relationship between TV distance and W1W_1 distance for finite metric spaces (Gibbs & Su, 2002, Thm. 4). Using this relationship, we obtain the following upper bound: pe≤pe∗+Cre+1dmin∑iPiW1(PX^i,ri)p_e\leq p_e^*+Cr_e+\frac{1}{d_{min}}\sum_{i}P_iW_1(P_{\hat X_i}, r_i) where dmind_{min} is the smallest distance between two distinct elements of the space. Since our images take values in a finite set, this new bound can be linked to our neural experiments.
  • Empirical transmitter error. The reviewer is right that 0.079 is not the Bayes error. Since the error of any transmitter-side test gives an upper bound for pe⋆p_e^\star, the upper bound in Thm. 1 would still be valid if we simply replaced pe⋆p_e^\star with the error of the transmitter-side test. We now state this in Thm. 1, and Sec. 7 and the figures now describe 0.079 as the transmitter-side classifier's error, a practical reference rather than the optimum.

W3 and Q2: quantizer settings for each figure. We thank the reviewer for pointing out this inconsistency. Figs. 5–7 use a vector quantizer with LL tokens and KK codewords, (L,K)=(8,8)(L, K) = (8, 8), (16,16)(16, 16), (16,64)(16, 64) and (32,16)(32, 16), that is 24, 64, 96 and 128 bits, plus H(J)=3.32H(J) = 3.32 bits when JJ is sent. Fig. 10 adds (8,16)(8, 16), (8,32)(8, 32), (8,64)(8, 64), (16,32)(16, 32) and (32,32)(32, 32), that is 32, 40, 48, 80 and 160 bits. Table 4 is replaced by a table containing every configuration trained, with R′R', R′+H(J)R' + H(J), the entropy measured on the codewords and the figures that use it.

AuthorsOpenReview ↗

Reply to Reviewer rbg8 (Part 2)

Q2: a label-only reference. We added a comparison of our scheme with several label-only schemes (App. H.3). In each, the transmitter sends only JJ (3.3 bits) and the receiver outputs an MNIST image of class JJ without using any other information about the input: a randomly chosen image of that class, one fixed image chosen for each class, or the average image of the class. A fourth scheme also sends the index of the MNIST image of class JJ that is closest to the input under our distortion (this requires the MNIST dataset to be available at the transmitter as well), which raises the rate to 11.5 bits.

SchemeRate (bits)Distortionpep_eW1W_1
JJ + random MNIST image of class JJ3.31.970.0850.06
JJ + one fixed MNIST image per class3.31.570.07913.1
JJ + mean MNIST image of class JJ3.31.550.53611.6
JJ + closest MNIST image of class JJ11.51.450.0916.0
Ours, send JJ, λ=10\lambda = 1027.31.300.1232.45

These label-only schemes reach the error floor at 3.3 bits, but their reconstructions are far from the input (distortion 1.55–1.97). Below a distortion of about 1.41.4, none of them approaches our codec, which reaches pe=0.12p_e= 0.12 at D=1.30D=1.30. Above about 1.451.45, sending a fixed image or the closest MNIST image matches our error at a lower rate, but those outputs are far from the MNIST distribution (W1W_1 between 6 and 13, against 1.0–3.7 for our codecs). The W1W_1 of the random-image scheme is optimistic, because the critic's reference images come from the same pool.

We thank the reviewer again and are happy to answer further questions.

References

  1. [Gibbs & Su(2002)] A. L. Gibbs and F. E. Su. Choosing and bounding probability metrics. International Statistical Review, 70(3):419–435, December 2002.
Reviewer_xH8rOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies lossy compression when reconstructions are processed by a fixed classifier whose decision regions may be mismatched with the source distribution. The authors formulate a rate–distortion–error problem and distinguish three levels of available domain information. They characterize the compression problem under known decision regions and derive a classification error bound based on class-conditional distribution matching when those regions are unknown. Discrete-alphabet experiments and neural compression experiments on SVHN-to-MNIST illustrate the relationship between reconstruction distortion, distribution matching, and classification performance.

Critical strengths
  1. A meaningful formulation of compression for a fixed, mismatched classifier.
  2. An interpretable sufficient condition connecting distribution matching to classification performance.
Critical weaknesses

1.The scope of the theoretical characterization and its connection to the neural implementation need qualification. The authors correctly acknowledge after Eq. (10) that the Level 2 distribution-matching constraint is sufficient rather than equivalent to the original classification-error constraint. However, the broader framing should distinguish this conservative surrogate from the Level 1 RDME characterization. In addition, Lemma 1 does not address posterior ties: Eq. (5) remains valid for mappings satisfying Eq. (4), but without an additional assumption, Eq. (4) does not characterize all mappings attaining the Bayes error. The resulting general H(J) lower bound therefore requires qualification. Finally, the neural objective uses Wasserstein-1 distance, whereas the error guarantees rely on TV/KL divergence. The practical motivation is reasonable, but the established bounds do not directly cover this objective. These issues matter because they limit how broadly the theoretical conclusions can explain the reported neural results. The posterior-tie issue appears repairable and is not, by itself, decisive. 2.The experimental resource assumptions and rate configurations are not sufficiently consistent to verify the central comparisons. Level 3 is described as providing only mixture distributions, yet the distortion measure uses a supervised SVHN classifier, and its feature layer is selected using class separation. Such prior information may be permissible, but it should be explicitly included in the information model. The rate configurations are also inconsistent across the experimental descriptions. Figures 5–7 show rates labeled 24, 64, 96, and 128 bits/image, with Figure 5 reporting R′R' before the additional cost of transmitting JJ. Table 4 instead lists 24, 32, 80, and 192 bits/image, while the reproduction command specifies configurations corresponding to 40, 48, 64, 96, and 128 bits/image. Because rate is a central comparison variable, these discrepancies impede verification of the compression results. Although the cost of transmitting J is acknowledged, comparisons at matched total rates and comparable distortion would better establish its benefit.

Questions for authors

1.Can you clarify which perception-based formulations are exact characterizations and which are sufficient surrogates? In the discrete experiments, how conservative is the Level 2 surrogate relative to the original classification-error constraint? Please also clarify the assumption needed to handle posterior ties in Lemma 1 and its consequences for the general H(J) rate lower bound. 2.Is the Wasserstein-based neural objective intended as a practical surrogate outside the TV/KL guarantees, or can a corresponding error bound be established under additional assumptions? A clear statement of which guarantees apply to the neural experiments would help resolve this concern.

2 comments
AuthorsOpenReview ↗

Reply to Reviewer xH8r (Part 1)

We thank the reviewer for a careful reading, and for noting the missing posterior-ties assumption. We have revised the paper accordingly to address the reviewer's concerns. All changes described below will appear in the revised PDF, which we will upload by November 15.

W1 and Q1: which formulations are exact, and which are sufficient surrogates.

  • Under the zero posterior-ties (ZPT) assumption, which we state below, Lemma 1 and (5) are exact characterizations.
  • Under the ZPT assumption, (7) recovers the RDME problem as P→∞P \to \infty in the discrete domain (q(x)>0∀x∈Xq(x)>0 \forall x\in \mathcal{X} is assumed). In the continuous domain, it only forms a sufficient surrogate that achieves the optimal error.
  • Thm. 1 and Cor. 1: These are rigorous upper bounds on pep_e that do not rely on the ZPT assumption.
  • Since R(D,α)≤R(2)(D,α)R(D, \alpha) \le R^{(2)}(D, \alpha), the RDP formulation in (10) is a sufficient surrogate for the RDE formulation in (2).
  • The RDP formulation in (11) is only a heuristic with no error guarantee. The KL divergence in the perception constraint may be replaced by another perception measure where appropriate.
  • The neural objective is a practical surrogate and can now be linked to our theory with the newly added Corollary 2 (see W1 and Q2 below).

W1 and Q1: Posterior ties. We thank the reviewer for raising this crucial concern about Lemma 1. Lemma 1 was derived under the assumption that P(πipi(X)=πjpj(X))=0P(\pi_i p_i(X) = \pi_j p_j(X))=0 for i≠ji \neq j (ZPT assumption), which is why, in the proof of Lemma 1, we took the function lj,i=πipi(x)−πjpj(x)l_{j,i}=\pi_i p_i(x) - \pi_j p_j(x) to be strictly negative in region Rj∗R_j^*. This assumption was missing from our original text, and we thank the reviewer for pointing this out. Under this assumption, Lemma 1 and (5) hold as stated; all our examples satisfy it. Without this assumption, the condition in Lemma 1 is sufficient to achieve the Bayes-optimal error, but not necessary, since a point with a posterior tie may be mapped into the region of any of its tied classes without raising the error. Further, under the ZPT assumption, the lower bound H(J)H(J) is still valid for the RDME formulation. In the absence of it, one may partition the tied points based on the tied classes and encode this information in JJ (in this case, X\mathcal{X} is partitioned into more than MM regions that take posterior ties into account, and each region takes a unique value of JJ). Given JJ, the stochastic mappings pX^∣Xp_{\hat{X}|X} then differ significantly across the tied regions and thus must be handled carefully.

Q1: How conservative is the Level 2 surrogate in the discrete example? We solved the programs (2) and (10) on the grid of Fig. 4 (Table 1 distributions, pe∗=0.12p_e^* = 0.12, re=0.03r_e = 0.03, C=1C = 1) for Pα=0.02P_\alpha = 0.02, 0.050.05 and 0.080.08:

  • (10) becomes feasible starting from D≈0.44D \approx 0.44–0.500.50; the feasible region of (2) starts from D≈0.12D \approx 0.12–0.220.22.
  • For the same value of α\alpha, (10) needs 0.38–0.79 more bits than (2).
  • However, the solutions of (10) reach pe≈0.16p_e \approx 0.16–0.200.20, well below α\alpha. If we instead constrain the program in (2) to the error that the solution of (10) actually reaches, at the same DD, it needs 0.08–0.39 fewer bits than (10), and this difference shrinks as DD grows.
AuthorsOpenReview ↗

Reply to Reviewer xH8r (Part 2)

W1 and Q2: the Wasserstein distance and the guarantees. We have now added a new corollary (Corollary 2) to Thm. 1 using the relationship between the TV distance and the W1W_1 distance that holds in finite metric spaces (Gibbs & Su, 2002, Thm. 4). Using this relationship, we obtain the following upper bound: pe≤pe∗+Cre+1dmin∑iPiW1(PX^i,ri)p_e\leq p_e^*+Cr_e+\frac{1}{d_{min}}\sum_{i}P_iW_1(P_{\hat X_i}, r_i) where dmind_{min} is the smallest distance between two distinct elements of the space. Since our images take values in a finite set, this new bound can be linked to our neural objective.

W2: information used at Level 3. We completely agree with the reviewer's concern about the information leakage present in Level 3 results. Level 3 assumes that only unlabeled source samples are available. However, since the Level 3 distortion function used features of the label-trained gXg_X, with the layer chosen by class separation, some class information may leak into the distortion function. To rectify this issue, we re-ran Level 3 with a label-free distortion metric that uses features from a ResNet trained with an unsupervised learning framework (SimCLR) (Chen et al., 2020). The table below shows these new results; we report pep_e at R=128R=128 bits.

DistortionLabels usedLevel 3Level 2, no JJ
gXg_X features (paper)yes0.3470.228
SimCLR featuresno0.4420.246
Edge map (Sobel)no0.7960.317
Pixel MSEno0.8000.297

Without labels, Level 3 still works when the distortion is learned from unlabelled source images, which Level 3 permits. The SimCLR features give pe=0.442p_e=0.442, compared with pe=0.347p_e=0.347 using gXg_X and pe=0.90p_e=0.90 without distribution matching. Fixed distortions (pixel MSE, Sobel) are not enough at Level 3: with pixel MSE, the outputs look like MNIST digits but are mostly of the wrong class (pe=0.80p_e=0.80). At Level 2 the class-conditional distribution already carries the class information, so even pixel MSE reaches pe=0.297p_e=0.297. In short, Level 3 needs a distortion that separates the classes, and labels are not needed to get one. App. H.4 gives the details.

W2: rate configurations. We thank the reviewer for pointing out this inconsistency. Figs. 5–7 use (L,K)=(8,8)(L, K) = (8, 8), (16,16)(16, 16), (16,64)(16, 64) and (32,16)(32, 16), that is R′=24,64,96R' = 24, 64, 96 and 128128 bits, plus H(J)H(J) when JJ is sent. Table 4 is replaced by a table containing every configuration trained, with R′R', R′+H(J)R' + H(J), the entropy measured on the codewords (always below the nominal rate, so the plotted rates are upper bounds) and the figures that use it.

W2: JJ against no JJ at matched total rate and comparable distortion. We added a learned entropy model to the loss, so the rate is minimized and measured by entropy coding, and the total rate with JJ includes the H(J)=3.32H(J)=3.32 bits for the label. The training loss is now the distortion, plus λ\lambda times the perception term, plus a fixed coefficient (β\beta) times the estimated rate in bits. This rate coefficient β\beta sets the trade-off between bits and distortion: the larger it is, the fewer bits the trained codec uses. We trained a separate codec for each of seven values of β\beta, from 0.0003 to 0.03, which spreads the codecs from a few bits up to about 170 bits in total. Averaged over three seeds, sending JJ gives a lower pep_e than not sending it at every value of the rate coefficient β\beta, for λ>0\lambda>0 and at comparable total rate. Between 14 and 70 bits, the difference in pep_e is 0.08–0.15 at λ=10\lambda = 10 and 0.14–0.18 at λ=20\lambda = 20, against a seed standard deviation of at most 0.04. At λ=10\lambda = 10, sending JJ at 16 bits in total gives pe=0.100p_e=0.100, while without JJ the best error at any rate up to 166166 bits is pe=0.231p_e=0.231. At equal total rate, sending JJ also gives a lower distortion, by about 0.2 at λ=10\lambda = 10 (for example 1.27 against 1.47 at 32 bits and λ=10\lambda = 10). Below about 12 bits the codec without JJ is unstable: one of three seeds collapses to pe=0.9p_e=0.9 at the two largest values of the rate coefficient β\beta, where the codec uses the fewest bits, which does not happen when JJ is sent. App. H.2 gives the details.

We hope this resolves the reviewer's concerns and are glad to clarify further.

References

  1. [Gibbs & Su(2002)] A. L. Gibbs and F. E. Su. Choosing and bounding probability metrics. International Statistical Review, 70(3):419–435, December 2002.
  2. [Chen et al.(2020)] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Summary of revisions and new experiments (Part 2)

New experiments (new App. H)

  • Level 3 without labels. To remove the label information that leaked into Level 3, we replaced the label-trained gXg_X features in the distortion metric with features from an encoder trained with SimCLR on the same SVHN images without labels. With these features Level 3 reaches pe=0.442p_e=0.442 compared with pe=0.347p_e=0.347 using the gXg_X features. The error is higher, but matching onto the target still lowers it substantially, and the overall trends remain the same.
  • Entropy-coded rate and JJ at matched total rate. We added a learned entropy model to the training loss, so the rate is now minimized and measured by entropy coding instead of being fixed by the quantizer. With a learned entropy model in the loss, there is a substantial trade-off between rate and distortion only below about 30 bits. Whenever the matching term is active (λ>0\lambda > 0), sending JJ gives a lower error than not sending it at a comparable total rate. For λ=10\lambda = 10, sending JJ at 16 bits in total achieves pe=0.100p_e= 0.100, which beats not sending JJ at every rate up to 166 bits, where the best is pe=0.231p_e=0.231. These numbers are averages over three seeds.
  • Label-only references. A trivial scheme that sends JJ and outputs a random MNIST image of class JJ reaches pe=0.085p_e=0.085 with only 3.3 bits, but its reconstruction has little to do with the input, resulting in a distortion of 1.97. Our send-JJ codec reaches pe=0.12p_e=0.12 with distortion 1.30, so it keeps the reconstruction closer to the source. A decoder that receives JJ but is trained without the matching term (without the perception term) performs no better than chance (pe=0.90p_e=0.90), so it is the matching, not JJ alone, that lowers the error.
  • Compress-then-translate baselines. We compared our codecs against compression followed by domain-transfer methods such as CycleGAN, CyCADA and PixelDA. In all 48 comparisons at the same total rate, our codecs reach a lower pep_e at a lower distortion.
  • Reporting. We fixed several inconsistencies in how the results were reported:
    • Rates. Table 4 did not match the configurations used in the figures; it now lists every (L,K)(L,K) we trained, the figures that use it and, next to each nominal rate Llog⁡2KL\log_2K, the entropy actually measured on the codewords generated by our trained encoder. The measured entropy is always below the nominal rate, so the rates in the figures are upper bounds.
    • Cost of JJ. We now report the measured entropy of the transmitted label, H(J)=3.3207H(J)=3.3207 bits, instead of assuming log⁡2M=3.3219\log_2M=3.3219 bits.
    • Seeds. We report the standard deviation over seeds for the neural results.

We hope these changes address the reviewers' concerns and welcome further questions during the discussion.

References

  1. [Gibbs & Su(2002)] A. L. Gibbs and F. E. Su. Choosing and bounding probability metrics. International Statistical Review, 70(3):419–435, December 2002.
AuthorsOpenReview ↗

Summary of revisions and new experiments (Part 1)

We thank the four reviewers for careful and constructive reviews. All four found the setting well motivated, and found the error bound of Thm. 1 interpretable. Three concerns were shared: (1) which results are exact characterizations and which are sufficient conditions or heuristics; (2) how the W1W_1-trained neural codecs relate to the TV/KL guarantees; (3) experimental reporting, including the information used at Level 3 and the rate configurations. We revised the paper accordingly (changes in blue) and ran the experiments below. Each reply gives the details. All changes described in these replies will appear in the revised PDF, which we will upload by November 15; we will then post a short note listing where each change appears.

Theory and framing

  • Zero posterior-ties (ZPT) assumption: We now state the main assumption under which Lemma 1 was derived. Let Aij={x∈X:πipi(x)=πjpj(x)}A_{ij}=\lbrace x\in\mathcal{X}: \pi_i p_i(x)=\pi_jp_j(x)\rbrace. We assume that AijA_{ij} has zero measure for 0≤i,j≤M−10\leq i,j\leq M-1 and i≠ji\neq j. This assumption has now been added to the main text.
  • Equivalence and sufficiency of RDP formulations: Under the ZPT assumption, (5) is exact and the H(J)H(J) lower bound holds. In the discrete domain, (7) is equivalent to the original RDME formulation as P→∞P\to\infty under the ZPT assumption. In the continuous domain, it is only a sufficient surrogate for achieving the optimal error probability. Thm. 1 and Cor. 1 are rigorous upper bounds and do not require the ZPT assumption. Therefore, (10) is a sufficient surrogate of the RDE formulation in (2) whereas (11) is a heuristic without an error guarantee. The introduction and conclusion no longer suggest an exact equivalence between the RDE and RDP formulations.
  • Thm. 1 holds for any transmitter-side test: Since the error of any transmitter-side test gives an upper bound for pe⋆p_e^\star, the upper bound in Thm. 1 would still be valid if we simply replaced pe⋆p_e^\star with the error of the transmitter-side test. We have now incorporated this adjustment into the upper bound in Thm. 1. The 0.079 test error reported in Sec. 7 is the transmitter-side classifier's error, and it provides a suitable upper bound for the receiver's classification error.
  • W1W_1 distance and the guarantees: We have now added a new corollary (Corollary 2) to Thm. 1 incorporating the relationship between the TV and W1W_1 distances in finite metric spaces (if dmin⁡d_{\min} is the smallest distance between two distinct elements of the space, then dmin⁡ TV≤W1d_{\min}\,\mathrm{TV} \le W_1; Gibbs & Su, 2002). Therefore, the presented neural experiments can now be directly linked to the theory through this newly added Corollary 2.
AuthorsOpenReview ↗

Follow-up to all reviewers: revised PDF

We have uploaded the revised PDF, with all changes marked in blue. As promised in our general comment, we list below where each change described in our replies appears. Section, equation, figure and table numbers refer to the revised PDF. All equation, figure and table numbers from the submission are unchanged; the new results are in the new Appendix H (Tables 7–11 and Figs. 14–18).

Theory and framing

ChangeWhere in the revised PDF
Zero posterior-ties (ZPT) assumptionAssumption 1 (Sec. 3, before Lemma 1); used in Lemma 1, at the start of Sec. 4, after (6), after (10), in Sec. 6, and in the proof of Lemma 1 (App. B)
Exact characterizations, sufficient surrogates and heuristicsIntroduction, paragraph after the three information levels; Sec. 4, paragraph after (7); Sec. 5, first paragraph (Thm. 1 and Cor. 1 do not need the ZPT assumption) and after (11) ((11) is a heuristic)
Finite KL and target-support coverage: (7) recovers the RDME problem as P→∞P\to\infty in the discrete domain (with q(x)>0q(x)>0 for all xx), and is a sufficient surrogate in general, even without the ZPT assumptionSec. 4, paragraph after (7)
Thm. 1 holds for any transmitter-side testThm. 1 (Sec. 5), last sentence; used in Sec. 7, second paragraph
Corollary 2: error bound in terms of W1W_1 on finite metric spacesSec. 5, after Cor. 1; it follows from Gibbs & Su (2002), as stated after the corollary; linked to the neural experiments in Sec. 7, after (12)
Related work: target matching in Liu et al. (2022) and Nguyen et al. (2026)Sec. 2, last paragraph
Role of common randomnessSec. 3, paragraph after Def. 1; main steps of the coding scheme in App. A.3
Intuition for R(D)=H(J)+R′(D)R(D)=H(J)+R'(D) and for the three terms of Thm. 1App. H.5, with both equations and a label for each term
Future work on the continuous domain and Gaussian sourcesSec. 8 (Conclusions), last sentence

Experiments and reporting

ChangeWhere in the revised PDF
0.0790.079 described as the error of the transmitter-side classifier gXg_X, a practical reference rather than the Bayes errorSec. 7, second paragraph; caption of Fig. 5 (Figs. 14 and 15 refer to it)
Level 3 without labels (SimCLR features, edge map, pixel MSE; Level 2 without JJ for comparison)Sec. 7, paragraph on Level 3; App. H.4 (Table 11, error and perception against λ\lambda in Fig. 17, reconstructions in Fig. 18); App. F.3 now states that the gXg_X features carry label information, which is Level 2 information
Entropy-coded rate, and JJ against no JJ at an equal total rate with H(J)H(J) includedApp. H.2 (Fig. 15, Tables 8 and 9); pointer in Sec. 7, after (12)
Label-only references, and the decoder that receives JJ but is trained for distortion onlyApp. H.3 (Table 10, outputs in Fig. 16; the same schemes also appear as markers in Fig. 14 of App. H.1)
Compress-then-translate baselines (CycleGAN, CyCADA, PixelDA)App. H.1 (Table 7 and Fig. 14). Our general comment said "at the same total rate"; the comparisons are at the same codec rate, and the configuration that sends JJ uses H(J)=3.32H(J)=3.32 additional bits
Rate configurationsTable 4 (App. F.2) now lists all nine (L,K)(L,K) that we trained, the figures that use each, and the measured entropy of the transmitted indices at λ=0,10,100\lambda=0, 10, 100; text in App. F.2 and Sec. 7
Cost of JJ: empirical H(J)=3.3207H(J)=3.3207 bitsSec. 7; App. F.2
Standard deviations over seedsApp. F.5 (seed spread of Figs. 5–7); App. H.1 (caption of Table 7); App. H.2 (Table 9, error bars in Fig. 15); App. H.4 (Table 11, error bars in Fig. 17)

App. F.5 now also describes how λ\lambda is scaled in the training objective, and App. F.3 has a rewritten sentence on the choice of the feature layer.

Numbers updated with more seeds

After posting our replies, we completed more seeds for the new experiments: the label-free runs (App. H.4) now use five seeds instead of two, and the entropy-coded runs (App. H.2) use five seeds instead of three. The revised PDF reports the five-seed results. The error probabilities change by at most 0.0140.014 and the distortions by 0.020.02, and none of the conclusions in our replies change. The results with the gXg_X features (four seeds), the label-only references and the baselines are unchanged. The entropy-coded values below can be read off Tables 8 and 9.

We thank the reviewers again and are happy to answer further questions during the remaining discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.