acceptodds
Under review as a conference paper at ICLR 2027

Scaling Context for Unified Variable-Rate and Variable-Complexity Image Compression

Abstract

Learned image compression (LIC) increasingly requires flexible rate–distortion–complexity (R–D–C) control for heterogeneous deployment. We introduce context scaling, a complementary scaling dimension that varies autoregressive context modeling within a fixed codec, rather than changing model capacity. By adjusting the context-modeling schedule, a single codec can jointly support variable rate and variable complexity while reusing the same transform output across complexity levels. Interestingly, we find that the benefit of context scaling is strongly input dependent: some images saturate with coarse context modeling, whereas others continue to benefit from finer-grained conditioning. Based on this observation, we introduce an adaptive context schedule that assigns finer context modeling to inputs with larger expected gains under a given complexity budget. Experiments on standard benchmarks show that context scaling provides effective R–D–C scalability, while adaptive scheduling consistently outperforms fixed schedules at matched average complexity. These results establish context modeling as a controllable complexity dimension orthogonal to model-capacity scaling, and reveal input-dependent context allocation as an effective mechanism for improving compression efficiency under constrained computation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Rate-distortion-complexity, Image coding, Variable-complexity, Autoregressive

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_2bSMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes context scaling for variable-rate and variable-complexity image compression. It adjusts autoregressive context steps and uses hyperprior information to allocate computation across images.

Critical strengths
  1. The problem is practical and worth studying.
  2. The image-dependent context gains are interesting.
  3. The experiments cover several datasets and recent baselines, with consistent improvements.
Critical weaknesses
  1. Changing AR context schedules is fairly straightforward. The novelty seems limited.
  2. The adaptive scheduler lacks comparisons with simple strategies, such as hyperprior-bitrate thresholding.
  3. MACs do not fully reflect the latency cost of sequential context modeling.
  4. The full complexity range actually relies on four separately trained models.
Questions for authors
  1. How does the scheduler compare with simple bitrate-based allocation?
  2. Why is non-uniform group merging better than uniform merging?
  3. Does the advantage remain under matched decoding latency?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 2bSM - Part I

Thank you for your thoughtful review. We address your concerns and questions below.

1. W1 & W4. What is new about changing context groups, and why four model capacities?

(a) Contribution and R–D–C performance. We agree that splitting or merging autoregressive (AR) groups is straightforward. Our contribution is not group merging itself, but establishing the granularity of conditional entropy modeling as a controllable computation dimension, characterizing its scaling behavior at both dataset and image levels, and using the observed image-dependent gains for rate–distortion–complexity (R–D–C) allocation. At a fixed rate condition, different context schedules reuse the same transform outputs, so changing context complexity does not require another transform pass.

This differs from capacity-based approaches such as slimmable models [1] or MoE-style routing, which primarily adjust the active transform capacity. RDCO [2] is a closer comparison within entropy modeling: it chooses where to apply spatial autoregressive context, using hyperprior-only or full-context prediction at different positions. ContextScale instead varies how finely latent dependencies are modeled across sequential groups. Both control entropy-model computation, but along different dimensions.

For clarity, we summarize the R–D–C comparisons in Fig. 4 of the main paper below, including the fixed-schedule and oracle references.

Table 1. PSNR BD-rate versus VTM-22.0 (%) at matched average complexity budgets.

DatasetBudget (kMAC/px)HPCM-VR-DMHPCM-SlimmableHPCM-RDCOContextScale (fixed)ContextScale (ours)ContextScale (oracle)
Kodak450−10.38−4.5−8.2−9.9−11.40−11.7
CLIC450−8.54−3.4−7.0−8.1−10.39−10.8
Tecnick450−11.66−6.7−10.5−11.2−13.56−13.7
Kodak900−15.25−8.8−12.1−14.8−15.94−16.4
CLIC900−14.18−7.9−11.3−13.9−15.21−15.7
Tecnick900−17.86−12.0−15.2−17.9−18.84−19.6
Kodak1350−17.53−11.9−15.0−17.3−17.96−18.2
CLIC1350−16.67−11.4−14.3−16.7−17.40−17.7
Tecnick1350−21.04−15.9−18.7−21.1−21.88−22.2

ContextScale achieves lower BD-rates than the listed baselines at the reported budgets. More importantly, adaptive scheduling improves over ContextScale (fixed) wherever both are compared, even though they use the same model and schedule family. Rather than assigning the same schedule to every image, it allocates more context computation to images that still benefit from finer conditioning (see also Appendix D.1 for the resulting schedule distributions).

(b) Why four capacities? Each ContextScale model supports multiple rates and context complexities within a single set of weights (see Appendix B.3, Table 3). We use four model capacities to efficiently cover a broad R–D–C range, but context scaling itself does not require multiple models.

Table 2. Complexity coverage within each model (reference forward kMACs/pixel).

ModelParametersSupported range
Small79.84M353.98–726.55
Base87.36M541.74–914.30
Large113.31M761.23–1257.84
Ultra135.83M1030.07–1650.74

In principle, a single model can support a wider complexity range by allowing more AR steps. For example, extending the Small-model configuration from 15 to 32 steps would expand its complexity range from 353.98–726.55 to 353.98–1342.77 kMACs/pixel, without increasing its transform capacity.

However, although additional AR steps enable finer conditional modeling, all context schedules share the same transform outputs. Consequently, the compression gains from additional steps gradually diminish, potentially reflecting the limitations of the shared latent representation. This motivates us to consider parameter scaling and context scaling as complementary dimensions, rather than relying solely on increasing context steps.

AuthorsOpenReview ↗

Response to Reviewer 2bSM - Part II

(c) Can the two dimensions be combined? Yes. As Fig. 2 suggests, parameter and context scaling are complementary. We also explored combining MoE-style capacity routing with ContextScale: MoE adjusts active transform capacity, while ContextScale adjusts conditional entropy-modeling computation.

Table 3. Exploratory comparison of capacity and context scaling.

Metric / target budgetContextScale-only (reference)MoE-onlyMoE + ContextScale
450 kMAC/px · ΔBDR (pp)0.00+5.80+1.19
900 kMAC/px · ΔBDR (pp)0.00+6.74+0.86
1350 kMAC/px · ΔBDR (pp)0.00+6.28+1.35
Complexity range (kMAC/px)353.98–1650.74454.62–1520.31408.50–1562.73
Number of models411
Complexity controlContext schedulesActive capacity / expertsCapacity + context schedules

ΔBDR is relative to ContextScale-only at the corresponding budget (lower is better). The ContextScale-only range is the combined coverage of four models; each MoE variant uses one model.

Adding context scaling to MoE reduces the BD-rate gap by 4.61, 5.88, and 4.93 pp at the three budgets, respectively. Although this exploratory single-model combination still trails the four-model ContextScale-only reference by around one percentage point, it shows that context scaling can complement capacity routing rather than replace it.

2. W2 & Q1. Does the scheduler beat simple hyperprior-bitrate thresholding?

We compared Single-threshold (two schedules) and Multi-threshold (multiple schedules) policies calibrated using rzr^z on held-out images, under matched average MAC budgets. While both baselines use the hyperprior bitrate, our scheduler estimates the expected R–D gain of each candidate schedule and weighs it against its computational cost.

Table 4. Threshold policy minus ContextScale (ours), BD-rate difference (pp).

DatasetSingle-thresholdMulti-threshold
Kodak+0.74+0.32
CLIC+0.61+0.24
Tecnick+0.56+0.12
Avg.+0.63+0.23

Positive differences favor ContextScale (ours).

Using multiple thresholds narrows the gap considerably, as several cutoffs offer more flexibility than a binary coarse/fine choice. ContextScale (ours) nevertheless retains an approximately 0.23-pp average advantage over Multi-threshold. This suggests that estimating the gain of each schedule, rather than only ranking images by rzr^z, can allocate computation more effectively. The threshold results also reinforce our observation in Fig. 3 that hyperprior bitrate is a useful indicator of context-scaling gains.

3. Q2. Why use non-uniform rather than uniform group merging?

Our motivation comes from the unequal importance of context groups during autoregressive coding. When adjacent groups are merged, they are coded in parallel and can no longer condition on one another. Since early decoded groups provide context for many subsequent predictions, we preserve finer partitions in early coding stages and merge later groups more aggressively, rather than merging all groups uniformly. This design is consistent with the uneven distribution of latent information observed in ELIC [3] and the importance of informative early groups emphasized by CCA [4].

To verify this design choice, we trained uniform-merging variants under comparable settings and compared both strategies at matched computational complexity.

Table 5. BDR(Non-uniform) − BDR(Uniform), percentage points.

Evaluated cost (kMAC/px)KodakCLICTecnickAvg.
385.98−0.73−1.55−1.63−1.30
471.13−1.39−2.94−3.07−2.47
556.27−1.44−2.55−2.96−2.32
641.41−0.88−1.54−1.80−1.41
Avg.−1.11−2.14−2.36−1.87

Negative values favor non-uniform merging.

Non-uniform merging consistently achieves lower BD-rate across all three datasets and four evaluated complexity levels, with an average improvement of 1.87 pp. These results support our design of preserving fine-grained conditional dependencies in early coding stages while reducing sequential computation through coarser grouping in later stages.

AuthorsOpenReview ↗

Response to Reviewer 2bSM - Part III

4. W3 & Q3. Does adaptation still help at matched decoding latency?

An important feature of ContextScale is that its adaptive scheduling framework is not tied to a specific complexity metric. Our method allocates context computation based on the expected R–D gain and computational cost of each candidate schedule. While we use kMACs/pixel in the main paper as a hardware-independent complexity measure, the same framework can accommodate other measurable or estimable costs, such as decoding latency, without changing the underlying scheduling principle.

To verify this, we conducted additional experiments using measured decoding latency as the complexity budget, including neural inference and CPU rANS coding at native image resolutions. We compare our adaptive scheduling strategy with fixed context schedules under matched average decoding latency.

Table 6. ContextScale (ours) versus fixed schedules at matched full decoding time.

DatasetMatched decoding time (ms)BDR(Fixed) − BDR(ContextScale) (pp)
Kodak66.8+1.19
Kodak71.2+1.08
Kodak85.7+1.03
Tecnick289.6+1.27
CLIC310.9+1.30
CLIC428.6+1.45
Avg.—+1.22

At all six operating points, ContextScale achieves 1.03–1.45 pp lower BD-rate than fixed scheduling, with an average improvement of 1.22 pp (see Appendix D.2, Fig. 11). Importantly, these results demonstrate that the benefit of our adaptive context-scaling strategy is not limited to MAC-based complexity budgets. The same gain-aware allocation framework remains effective under measured latency constraints, further supporting its flexibility for different computational requirements.

Closing

To summarize, we believe ContextScale makes several distinct contributions. To our knowledge, it is the first unified variable-rate and variable-complexity LIC framework to achieve performance competitive with conventional state-of-the-art LIC methods. We introduce and systematically study context scaling as a new complexity-scaling dimension, and further investigate image-wise scaling behavior to develop an adaptive strategy that allocates computation according to input-dependent gains. Importantly, context scaling is complementary to parameter scaling, generalizes across different conditional entropy models (e.g., HPCM, CHARM, and Checkerboard), and is not restricted to a particular complexity metric, as demonstrated using both kMAC-based and measured-latency budgets.

We have incorporated part of the additional results and analyses into the Appendix, highlighted in blue, and plan to integrate the remaining findings into the main paper following further discussions and peer review. We sincerely appreciate your constructive feedback, which has helped us better clarify and strengthen the contributions of our work.

References

[1] F. Yang, L. Herranz, Y. Cheng, and M. G. Mozerov, “Slimmable Compressive Autoencoders for Practical Neural Image Compression,” CVPR, 2021, pp. 4996–5005.

[2] Y. Gao, R. Feng, Z. Guo, and Z. Chen, “Exploring the Rate-Distortion-Complexity Optimization in Neural Image Compression,” Journal of Visual Communication and Image Representation, vol. 105, art. 104294, 2024.

[3] D. He, Z. Yang, W. Peng, R. Ma, H. Qin, and Y. Wang, “ELIC: Efficient Learned Image Compression With Unevenly Grouped Space-Channel Contextual Adaptive Coding,” CVPR, 2022, pp. 5718–5727.

[4] M. Han, S. Jiang, S. Li, X. Deng, M. Xu, C. Zhu, and S. Gu, “Causal Context Adjustment Loss for Learned Image Compression,” NeurIPS, 2024.

Reviewer_3TDFOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes context scaling, which treats the number of autoregressive entropy-modeling steps as a complexity dimension orthogonal to model capacity, letting a single codec support both variable rate and variable complexity while reusing the transform. The authors find that the scaling gain is input-dependent and predictable from the hyperprior bitrate, and design a budget-constrained adaptive context schedule accordingly.

Critical strengths
  1. Context scaling as a new complexity dimension has a genuine empirical basis. The power-law fit reaches R² of 0.997, the orthogonality to parameter scaling is visible in the iso-KMAC contours, and transform reuse has direct engineering value for content distribution scenarios.
  2. The adaptive scheduling turns the observation into a deployable mechanism. Hyperprior-based gain prediction requires no extra inference, online debt control keeps the streaming budget on target, it consistently improves over fixed schedules while staying close to the oracle, and it generalizes to structurally different context models such as CHARM.
Critical weaknesses
  1. All results are built on the single HPCM backbone. Although CHARM is used in an ablation, the training scale, hyperparameters, and schedule design are tuned for HPCM, and extrapolation to non-autoregressive or fully parallel entropy models is unknown.
  2. A systematic analysis of joint rate and complexity variation is missing. Current results mostly vary complexity at fixed rates, and the interaction between variable rate and variable complexity, such as whether the optimal schedule is consistent across rates, is under-discussed.
Questions for authors
  1. Can context scaling be applied directly to non-autoregressive parallel entropy models such as checkerboard or purely channel-wise context models without redesigning the schedule?
  2. Is the optimal context schedule consistent across different rate conditions, and is there a coupling effect between the rate and complexity conditions?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 3TDF

Thank you for the encouraging assessment and for the two useful questions. We examined transfer across context structures and the interaction between rate and context scheduling more closely.

1. W1 & Q1. Can ContextScale work beyond HPCM's context structure?

Yes, for entropy models that exploit conditional dependencies across decodable groups. Conditional entropy modeling is widely used in modern LIC, although the context structures vary across architectures. ContextScale builds on this common principle rather than a specific spatial or channel-wise design. In the main paper, we already demonstrated its applicability to CHARM's channel-wise context model [1], while retaining the HPCM transform backbone for a controlled comparison. To further address your question, we conducted new experiments with a checkerboard spatial context model [2], which enables parallel prediction within spatial groups. The extended results are included in Appendix D.3 (Table 4).

Table 1. Transfer across conditional context structures (average PSNR BD-rate, %).

Context modelBaseline-VR+ContextScaleContextScale (oracle)
CHARM−9.77−11.04−12.63
Checkerboard−8.65−10.10−11.82

ContextScale improves BD-rate by 1.27 pp with CHARM and 1.45 pp with Checkerboard. These models organize conditional dependencies differently: CHARM uses channel groups [1], Checkerboard uses spatial groups [2], and HPCM combines spatial and channel-wise context. Nevertheless, the underlying ContextScale framework and adaptive scheduling principle remain unchanged. Only the group partition and coding order need to be adapted to the corresponding entropy model; no redesign of the allocation strategy is required. Thus, ContextScale is not tied to HPCM's particular context structure, but can generalize to different conditional entropy models, including those with parallel prediction within groups.

2. W2 & Q2. Does the optimal schedule depend on rate?

Yes, for some images; the effect is not uniform. We examined this from both the offline oracle and practical scheduler perspectives.

(a) Oracle assignments. The oracle chooses schedules using the measured R–D gains of all candidates under a complexity budget. Comparing its assignments across the rate conditions q∈{16,32,48}q\in\{16,32,48\} gives the following fractions of images assigned different schedules:

Table 2. Images assigned different oracle schedules across the tested rate conditions.

KodakCLICTecnick
29.79%27.99%23.00%

This shows that rate can change the preferred context allocation, although many images retain the same schedule across the tested conditions.

(b) Practical scheduling. Unlike the Oracle, our practical scheduler cannot access the actual R–D gains of all candidate schedules. Instead, it estimates their expected gains using the rate condition qq and hyperprior bitrate rzr^z (Sec. 4.2 and Appendix A.3), and selects a schedule under the target complexity budget.

Interestingly, rzr^z already captures information about both image content and the rate condition, as the hyperprior representation and its bitrate vary with qq. To examine this, we tested a variant that removes explicit qq conditioning from the gain predictor, using only rzr^z for gain estimation while retaining the same budget-control framework. This variant showed only a small BD-rate loss of approximately 0.2% relative to the original scheduler.

Together with the Oracle analysis, these results suggest that the optimal schedule can vary with rate, but much of this interaction can already be captured through rzr^z. Our original method retains explicit rate-conditioned calibration; this ablation simply examines its contribution to gain prediction.

Closing

Thank you for your thoughtful comments and valuable discussion. We hope our responses have clarified the key contributions of ContextScale: introducing context scaling as a complementary dimension to parameter scaling, revealing image-dependent scaling behavior and exploiting it through adaptive scheduling, and extending this principle across different conditional entropy models.

During the rebuttal, we further extended our experiments from kMAC-based complexity budgets to measured decoding latency, where adaptive scheduling consistently outperforms fixed schedules. This provides additional evidence that our framework is not restricted to a particular context structure or complexity metric. We have incorporated part of these additional results into the Appendix and welcome any further comments or suggestions.

References

[1] Minnen et al., “Channel-Wise Autoregressive Entropy Models for Learned Image Compression,” ICIP.

[2] He et al., “Checkerboard Context Model for Efficient Learned Image Compression,” CVPR.

As the rebuttal period is coming to an end, we hope our responses have addressed all your concerns. Thank you for your positive feedback and support! We'd be happy to discuss any remaining questions.

Reviewer_hkbFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper makes the autoregressive context schedule (the number of sequential latent groups) a complexity knob inside a single HPCM-based codec, which it calls "context scaling". Rate and schedule are injected through AdaLN embeddings. The schedules form a nested family obtained by merging groups of the finest partition, and rate-schedule pairs are sampled jointly during training, so the analysis transform is shared across complexity levels. The authors fit power laws of BD-rate in the number of steps S and in model size N. They observe that the per-image benefit of finer context varies widely and correlates with hyperprior bitrate (Spearman rho = 0.84). On this basis they propose an adaptive scheduler: a 6-bin lookup of expected gain given r^z, a Lagrangian choice of schedule, and a debt-based controller that targets an average complexity budget over a stream. Results include gains over model-size scaling at matched kMACs/px (e.g. -11.78% vs -10.19% at 450), proximity to an oracle knapsack allocation, comparisons with slimmable and RDCO variants on the same backbone, and a transfer to CHARM.

Critical strengths
  1. The design is simple and practical. Confining the complexity knob to the entropy model lets the analysis transform be shared across levels, and per-image selection from r^z costs almost nothing. The variable-complexity baselines are reimplemented on the same backbone, which makes the rate-distortion-complexity comparison fair.

  2. The finding that the benefit of context varies a lot across images and can be predicted from r^z is useful. It is checked across model sizes, QPs and schedules (Fig. 8), and the oracle allocation gives a meaningful upper bound.

Critical weaknesses
  1. The main claims are stated more strongly than the evidence supports. The scaling-law fits use a three-parameter form on 5 points (S = 4-8) and on 4 model sizes within less than a 2x range. An R^2 of 0.98-0.997 is expected in that setting, and nothing is predicted out of sample. More importantly, the key claim (adaptive scheduling beats fixed schedules at matched complexity and comes close to the oracle) is shown only in plots. Table 1 compares against HPCM-VR-DM, which uses a different model per budget, not against ContextScale with a fixed schedule. It is also unclear whether Table 1's 450 / 900 / 1350 rows come from one model or from different model sizes, which affects the "unified codec" claim. Since the size of the adaptive gain is the paper's central contribution, not being able to read it off is a significant gap.

  2. The complexity budget does not reflect deployment cost, and the stream controller conflicts with the no-signaling claim. Finer schedules mainly add sequential steps (Fig. 5), so MACs are a poor proxy for cost. At a similar BD-rate, DCVC-UF-Intra (-11.66%) runs in 16.4 / 12.5 ms against 49.7 / 48.8 ms for ContextScale-450 (-11.78%). Device constraints are usually per-image latency caps, not stream averages. In addition, the chosen schedule depends on the accumulated debt d_t (Eqs. 26-28), which a decoder cannot reproduce under random access. The schedule must therefore be signaled, or the claim restricted to in-order decoding. Varying the number of context steps is also well studied (e.g. RDCO), so the conceptual contribution is narrower than presented.

Questions for authors
  1. Which model size produces each ContextScale row in Table 1, and what complexity range does each model cover on its own?
  2. How is the schedule recovered at the decoder under random access?
  3. Can you provide a table of BD-rates for fixed vs adaptive vs oracle scheduling at matched budgets, per dataset?
  4. What does unification cost? Please compare the finest and coarsest schedules of the unified model with dedicated single-schedule models.
  5. Does r^z beat simple predictors (estimated total rate under the coarsest schedule, image size, random allocation), and does it correlate with relative, not only absolute, gains?
  6. How does adaptive scheduling compare with fixed schedules when latency, rather than kMACs, is the budget?
5 comments
AuthorsOpenReview ↗

Response to Reviewer hkbF - Part I

Thank you for the careful review. Your questions helped us identify where the original paper needed clearer evidence and explanations. We have run several additional experiments and address each point below.

1. W1. Do the scaling trends extend beyond the original points?

Our initial experiments focused on the model sizes and context complexities commonly used in learned image compression. We agree that the original four or five data points alone are insufficient to establish extrapolation. To test this directly, we kept the power-law fits from the original points fixed, rather than refitting them with the new measurements:

BDR⁡(u)=bu+auu−αu,u∈{N,S},\operatorname{BDR}(u)=b_u+a_u u^{-\alpha_u},\qquad u\in\{N,S\},

We then evaluated the predictions outside the original ranges, adding 50M- and 400M-parameter models and extending the tested context schedules to 1–16 autoregressive (AR) steps. On the held-out measurements, these fixed fits achieve R2=0.954R^2=0.954 for parameter scaling and 0.9660.966 for context scaling. While these are empirical trends rather than universal laws, the held-out measurements support their predictive value beyond the points used for fitting Fig. 2.

2. Q1. Which model produces each operating point, and how wide is its complexity range?

(a) Complexity coverage within one model. The 450, 900, and 1350 kMAC/pixel operating points in Table 1 of the main paper come from the Small, Large, and Ultra models, respectively. We should have distinguished the range of an individual model from that of the full model family more clearly. Within each model, however, different rates and context schedules are supported by the same weights. The model-specific ranges are also documented in Appendix B.3 (Table 3).

Table 1. Complexity coverage of each ContextScale model (kMAC/pixel).

ModelParametersSupported rangeBudget in main-paper Table 1
Small79.84M353.98–726.55450
Base87.36M541.74–914.30—
Large113.31M761.23–1257.84900
Ultra135.83M1030.07–1650.741350

(b) Why use different capacities? The use of four capacities is an efficiency choice, not an inherent requirement of context scaling. A single model can be designed to cover a wider complexity range by supporting additional AR steps. For example, a Small-model configuration with up to 32 steps would span 353.98–1342.77 kMAC/pixel, compared with 353.98–726.55 with up to 15 steps.

The trade-off is that, although increasing the number of AR steps enables finer conditional modeling, all context schedules share the same transform output. As a result, the compression gains from additional context steps gradually diminish, as the shared latent representation limits further improvements. This motivates us to consider parameter scaling and context scaling as complementary dimensions, rather than relying solely on increasing context steps.

(c) Can the two dimensions be combined? Yes. As Fig. 2 suggests, parameter scaling and context scaling are complementary rather than competing choices. We also explored combining MoE-style capacity routing with ContextScale. MoE adjusts active transform capacity, while ContextScale adjusts conditional entropy-modeling computation.

Table 2. Exploratory comparison of capacity and context scaling.

Metric / target budgetContextScale-only (reference)MoE-onlyMoE + ContextScale
450 kMAC/px · ΔBDR (pp)0.00+5.80+1.19
900 kMAC/px · ΔBDR (pp)0.00+6.74+0.86
1350 kMAC/px · ΔBDR (pp)0.00+6.28+1.35
Complexity range (kMAC/px)353.98–1650.74454.62–1520.31408.50–1562.73
Number of models411
Complexity controlContext schedulesActive capacity / expertsCapacity + context schedules

ΔBDR is relative to ContextScale-only at the corresponding budget (lower is better). The ContextScale-only range is the combined coverage of four models; the two MoE variants each use one model.

Adding context scaling to MoE closes the BD-rate gap by 4.61, 5.88, and 4.93 pp, respectively. Although this exploratory single-model combination still trails the four-model ContextScale-only reference by around one percentage point, it shows that context scaling can complement capacity routing rather than replace it.

AuthorsOpenReview ↗

Response to Reviewer hkbF - Part II

3. W1 & Q3. How much does adaptive scheduling improve over fixed schedules and existing methods?

We agree that the adaptive gain should be readable directly, not only inferred from Fig. 4. The table below includes HPCM-VR-DM (separate models at different budgets), HPCM-Slimmable (active-capacity scaling), HPCM-RDCO (spatial selection of hyperprior-only or full-context prediction), and three ContextScale variants: fixed (one schedule for all images), ours (content-adaptive), and oracle (offline allocation using measured candidate gains).

Table 3. PSNR BD-rate versus VTM-22.0 (%) at matched average complexity budgets.

DatasetBudget (kMAC/px)HPCM-VR-DMHPCM-SlimmableHPCM-RDCOContextScale (fixed)ContextScale (ours)ContextScale (oracle)
Kodak450−10.38−4.5−8.2−9.9−11.40−11.7
CLIC450−8.54−3.4−7.0−8.1−10.39−10.8
Tecnick450−11.66−6.7−10.5−11.2−13.56−13.7
Kodak900−15.25−8.8−12.1−14.8−15.94−16.4
CLIC900−14.18−7.9−11.3−13.9−15.21−15.7
Tecnick900−17.86−12.0−15.2−17.9−18.84−19.6
Kodak1350−17.53−11.9−15.0−17.3−17.96−18.2
CLIC1350−16.67−11.4−14.3−16.7−17.40−17.7
Tecnick1350−21.04−15.9−18.7−21.1−21.88−22.2

First, adaptive scheduling provides additional gains over fixed scheduling. Using the same model and candidate context schedules, our adaptive strategy improves BD-rate by 1.50/2.29/2.36 pp on Kodak/CLIC/Tecnick at 450 kMAC/pixel, and 1.14/1.31/0.94 pp at 900 kMAC/pixel, while remaining close to the oracle. These improvements come from allocating context computation according to image-dependent gains, rather than applying the same schedule to every image.

Second, ContextScale also outperforms existing complexity-scaling approaches. At 900 kMAC/pixel, it achieves −15.94/−15.21/−18.84% on Kodak/CLIC/Tecnick, compared with −12.10/−11.30/−15.20% for HPCM-RDCO, and consistently improves over HPCM-VR-DM at all three budgets. Unlike RDCO, which determines where spatial context is applied, ContextScale controls how finely conditional dependencies are modeled. Its contribution goes beyond group merging by revealing and exploiting image-dependent context-scaling gains for adaptive computation allocation.

4. Q4. What does unification cost relative to dedicated single-schedule models?

We trained dedicated models for the coarsest and finest schedules and compared them with the jointly trained ContextScale model under comparable capacity and training conditions.

Table 4. BDR(ContextScale) − BDR(Dedicated) (pp; negative values favor unified training).

Context scheduleBDR(ContextScale) − BDR(Dedicated)
Coarsest−1.12
Finest+0.56

Interestingly, joint training improves the coarsest schedule by 1.12 pp, while the dedicated model is 0.56 pp better at the finest schedule. One possible explanation is that training across schedules encourages the early decoded groups to remain informative for later predictions, even under coarse context modeling. This is consistent with the motivation of Causal Context Adjustment Loss (CCA; NeurIPS 2024), which emphasizes the role of informative early groups in predicting later groups. Unlike CCA, our method does not introduce an auxiliary loss but instead jointly trains different context schedules. At the finest schedule, a dedicated model can specialize more to that particular factorization. Thus, unification does not incur a uniform penalty; its effect depends on the schedule.

5. W2 & Q2. How does the decoder recover the schedule under random access?

The schedule is explicitly stored in a 32-bit field in each image header. The decoder reads that field and follows the indicated context schedule; it does not need the previous images or the encoder's accumulated complexity debt. The header was already included in the bitrates reported in the main paper, and the 32-bit field contributes less than 0.0001 bpp on our test images.

We also tested independent and out-of-order decoding at different context complexities; the reconstructions matched those obtained by sequential decoding. Our wording in the main-paper contribution—“no additional signaling overhead”—was imprecise: the gain predictor requires no extra model inference, but the selected schedule does require this small header field. We have clarified this distinction in Appendix B.1 and will correct the corresponding wording in the main paper.

AuthorsOpenReview ↗

Response to Reviewer hkbF - Part III

6. Q5. Why use hyperprior bitrate, and what about relative gains?

(a) Alternative allocation signals. We compared scheduling based on the realized hyperprior bitrate rzr^z with image-size-based allocation, random allocation, and allocation using the total rate under the coarsest schedule. All comparisons use matched schedule-execution complexity.

Table 5. Alternative minus ContextScale (ours), BD-rate difference (pp).

DatasetImage sizeRandom allocationCoarsest-schedule total rate
Kodak+1.07+1.00−0.25
CLIC+0.92+0.85−0.32
Tecnick+0.93+0.74−0.34
Avg.+0.97+0.86−0.30

Positive differences favor ContextScale (ours). The coarsest-schedule total-rate comparison excludes the extra computation needed to obtain that predictor.

As shown in Fig. 3, the benefit of additional context modeling varies considerably across images. Image size alone cannot capture these differences, while random allocation does not exploit the statistical properties of the latent representation. Using the coarsest-schedule total rate yields 0.30 pp lower BD-rate on average in this comparison. However, obtaining that signal requires an extra main-latent entropy-modeling/coding pass before the final schedule can be selected. In contrast, rzr^z is available from the hyperprior bitstream before main-latent coding, without that additional pass. We therefore favor a readily available signal, without claiming that rzr^z is the most informative predictor in isolation.

(b) Absolute versus relative R–D gain. In the paper, “relative to the coarsest schedule” identifies the reference schedule; it does not mean that the gain is percentage-normalized:

J_i(c;q)&=R_i(c;q)+\lambda(q)D_i(c;q),\\ \Delta_i(c;q)&=J_i(c_0;q)-J_i(c;q). \end{aligned}$$ A normalized relative gain would instead be $\Delta_i(c;q)/J_i(c_0;q)$. Since the baseline R–D loss varies across images, a larger percentage improvement does not necessarily mean a larger absolute gain. In the **Small/Kodak/$q=32$** setting, the Spearman correlation between $r^z$ and absolute gain is **0.8256**, compared with **0.2410** for normalized gain. Our scheduler aims to **maximize the total R–D loss reduction under a complexity budget**, making absolute gain the more appropriate allocation objective. The stronger correlation with $r^z$ further supports its use for estimating which images benefit most from additional context modeling (see **Appendix C.3, Fig. 9**, for additional predictor comparisons). > ### 7. W2 & Q6. Does the benefit remain under measured decoding latency? **(a) Separating backbone speed from context-scaling overhead.** You are right that ContextScale is slower than DCVC-UF-Intra in Table 1. Since the codecs use different backbones and CUDA implementations, this comparison reflects more than the cost of context scaling itself. A closer controlled comparison is **HPCM-VR-DM**, which shares our HPCM backbone: **Table 6. Same-backbone runtime and R–D comparison (runtime at 512 × 768 resolution; average BD-rate across Kodak/CLIC/Tecnick).** | Budget (kMAC/px) | HPCM-VR-DM enc./dec. (ms) | ContextScale enc./dec. (ms) | HPCM-VR-DM avg. BDR (%) | ContextScale avg. BDR (%) | |---|---|---|---:|---:| | 450 | 49.2 / 48.8 | 49.7 / 48.8 | −10.19 | **−11.78** | | 900 | 60.1 / 60.0 | 60.5 / 59.9 | −15.76 | **−16.66** | | 1350 | 72.8 / 72.3 | 69.9 / 69.2 | −18.41 | **−19.08** | At matched complexity budgets, the two methods have similar encoding and decoding times, while ContextScale achieves better average R–D performance. **(b) Using decoding latency as the budget.** We use kMAC/pixel in the main paper as a **hardware-independent measure of computational complexity**, although it does not directly reflect actual latency. Our scheduling framework is not restricted to MAC-based costs. We therefore also evaluated adaptive versus fixed scheduling at matched **measured decoding time**, including neural inference and CPU rANS coding on native-resolution images. **Table 7. Fixed versus adaptive scheduling at matched full decoding time.** | Dataset | Matched decoding time (ms) | BDR(Fixed) − BDR(ContextScale) (pp) | |---|---:|---:| | Kodak | 66.8 | +1.19 | | Kodak | 71.2 | +1.08 | | Kodak | 85.7 | +1.03 | | Tecnick | 289.6 | +1.27 | | CLIC | 310.9 | +1.30 | | CLIC | 428.6 | +1.45 | | **Avg.** | — | **+1.22** | Adaptive scheduling achieves **1.03–1.45 pp lower BD-rate** at these latency-matched points, confirming that its benefits extend beyond MAC-based comparisons. While our current formulation targets an **average complexity or latency budget**, per-image latency constraints can also be incorporated by restricting schedule selection to candidates whose estimated decoding latency is within the specified limit. The measured-latency evaluation is also presented in **Appendix D.2 (Fig. 11)**.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.