acceptodds
Under review as a conference paper at ICLR 2027

UniMAC: Unified Multi-modal Auxiliary Learned Image Compression

Abstract

Learned image compression methods typically operate solely on RGB data, limiting their ability to benefit from complementary information provided by other sensing modalities. Although recent approaches have explored auxiliary inputs such as infrared images, depth maps, point clouds, and stereo views, they generally require a dedicated architecture for each modality, making them difficult to scale and deploy across heterogeneous sensing scenarios. To address this limitation, we propose UniMAC (Unified Multi-modal Auxiliary Learned Image Compression), a unified framework that enables a single compression model to leverage diverse auxiliary modalities. We first employ modality-specific input heads to account for the distinct characteristics of heterogeneous sensor data and project them into a unified feature space. The resulting features are subsequently processed by a shared cross-sensor auxiliary feature extractor, which distills modality-agnostic representations relevant to image compression. These representations are then integrated into the primary RGB compression pipeline to improve its rate–distortion performance. Experiments across diverse cross-sensor settings demonstrate that UniMAC effectively exploits complementary auxiliary cues while avoiding the need to design or train a separate compression model for each modality.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Learned image compression, Multi-modality data compression, Inter-modality prediction, Inter-modality correlation utilization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Af8jOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes UniMAC, a unified framework for multi-modal auxiliary learned image compression. It enables a single shared-weight model to leverage diverse auxiliary modalities without separate codecs per scenario. The framework includes modality-specific input adaptation, shared multi-scale auxiliary extraction, auxiliary-conditioned compression, and a unified training strategy. Experiments across five multi-sensor scenarios show consistent rate-distortion improvements over baselines.

Critical strengths
  1. The proposed UniMAC accommodates multiple heterogeneous auxiliary modalities using a single shared codec, which is a meaningful direction for multi-sensor systems.

  2. The proposed UniMAC introduces modality-specific input adaptation, shared multi-scale auxiliary extraction, auxiliary-conditioned compression, and a unified multi-modal training strategy to effectively exploit heterogeneous multi-modal information.

Critical weaknesses
  1. The proposed UniMAC still relies on manually designed modality-specific stems and preprocessing pipelines. When introducing a new modality, explicit engineering is still required rather than direct plug-and-play generalization. Therefore, calling it "unified" may be imprecise.

  2. The paper lacks fine-grained ablation studies on the four core stages of the proposed framework, making it difficult to clearly distinguish the individual contribution of each stage to the overall performance gains.

  3. In real-world scenarios, auxiliary multi-modal information may be imperfect. For example, depth maps captured by consumer-grade sensors are often degraded, low-resolution, or sparse. It remains unclear whether the proposed method can maintain strong generalization and compression performance under such imperfect auxiliary inputs.

  4. The proposed framework appears to rely largely on conventional multi-scale and multi-modal feature fusion strategies. Its key methodological novelty and the distinction from existing fusion-based approaches should be further clarified.

  5. The rationale for using gated fusion in the first three stages and cross-attention only in the fourth stage (Eqs. 6–7) is unclear. This design should be better justified, and whether cross-attention at earlier stages could further improve performance is worth investigating.

  6. The comparison with only three competing methods is insufficient to fully demonstrate the effectiveness of UniMAC. Moreover, the qualitative results lack visual comparisons with existing methods.

Questions for authors

please see Weaknesses

Reviewer_fsDfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes UniMAC (Unified Multi-modal Auxiliary Learned Image Compression), aiming to enhance RGB image compression performance by leveraging auxiliary information from diverse sensors within a single shared-weight learning-based image compression framework. However, it assumes all other modalities are lossless, with both encoder and decoder endpoints aware of these lossless details. This assumption directly narrows the applicability of the entire methodology. Meanwhile, the authors identify gradient conflicts during training across different modalities but offer no solutions. Experimental results confirm the effectiveness of UniMAC's framework and training strategy in utilizing multimodal auxiliary image coding, though its lossless assumption and modal conflicts inherently limit its utility.

Critical strengths

UniMAC proposes a multi-modal condition-augmented RGB image compression method with shared weights. By designing a conditional network and employing a specific training strategy, the method enables the injection of conditional priors from diverse modalities, achieving significant performance improvements.

Critical weaknesses

The underlying assumption of UniMAC is that lossless acquisition of information from other auxiliary modalities at both the encoder and decoder ends can be leveraged to enhance lossy image coding; however, this significantly limits the applicability of the overall framework. While the identification of modality conflicts is a valuable finding, no robust solution for addressing this issue has yet been proposed.

Questions for authors
  1. This paper treats auxiliary modalities as known external information available at both the encoding and decoding ends. Please provide examples of specific application scenarios where this approach is employed, and also report the bitrate allocations for other modalities.
  2. The benefits of multi-stage auxiliary information fusion may stem from additional model capacity. The auxiliary information simultaneously influences the primary transformation, the entropy model, and the reconstruction refinement network. The authors need to demonstrate the contribution of each stage and quantify the specific gains achieved by the auxiliary information, given comparable parameter counts and inference costs.
Reviewer_bksgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

UniMAC studies RGB image compression when a registered auxiliary modality is available, independently transmitted, and perfectly known to both encoder and decoder (Section 3.1). It adds modality-specific stems, shared auxiliary feature extraction, and zero-initialized residual fusion to an HPCM codec, then jointly trains across stereo, depth, infrared, sparse point clouds, and segmentation maps (Figure 1; Sections 3–4). Table 1 reports bitrate savings relative to HPCM across all five scenarios. The paper also tests zero and shuffled auxiliary controls, single-modality training, auxiliary quality, and pairwise training interactions (Table 1; Figures 5–6; Appendix).

Critical strengths

1)A useful and clearly stated conditional-compression setting. The encoder/decoder side-information assumption is explicit (Section 3.1; Eq. 1), and the five auxiliary types cover genuinely different sensor/data forms (Section 4.1). This makes the scope of the contribution easier to judge than an unspecified “multimodal” claim. 2)Careful controls for the source of compression gains. Relative to HPCM, the reported BD-rate is −29.45% for stereo, −10.33% for depth, −7.78% for infrared, −51.97% for sparse point clouds, and −11.15% for segmentation (Table 1; Figure 2). Both objective curves and visual examples are provided (Figures 2–4).

Critical weaknesses
  1. The central auxiliary benefit is modest in several modalities.
  2. The “unified” model suffers negative transfer.
  3. The cost of side information is omitted.
  4. Architectural attribution and deployment cost are not fully resolved.
  5. Generalization beyond the training modalities is not demonstrated.*
Questions for authors
  1. In which use cases is the auxiliary stream already available at both endpoints, and how does total bitrate change when it must be transmitted?
  2. What is the average and worst-case performance penalty of the five-modality model relative to five independently trained variants, and how much model storage is saved?
  3. Are the small infrared and point-cloud improvements over UniMAC-Zeros stable across training seeds?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.