acceptodds
Under review as a conference paper at ICLR 2027

Where Should the Compute Go? Practical Neural Image Compression across Budgets

Abstract

Learned image compression has made substantial progress, yet its high computational cost remains a major barrier to practical deployment. Under tight complexity budgets, effective design requires not only reducing model complexity but also allocating computation across components. Taking edge-device complexity as the starting point, we present a systematic study of learned image compression under constrained budgets. Concretely, we first build a unified lightweight architecture and analyze computation allocation between the transform modules and the entropy model, then perform local structure search by independently optimizing and composing sub-modules, and finally construct a scalable model family via gradual expansion, with compact variants further improved by energy-based hierarchical pruning. The key aspect of our study is a systematic design procedure rather than any single network: under the unified architecture we establish, each specified complexity budget yields a configuration that consistently outperforms comparable low-complexity baselines across the evaluated constraints. As a representative result, with VTM-24.0 as the common anchor, our model achieves a 24.82% BD-rate reduction on Kodak relative to Shallow-NTC, a representative low-complexity decoding baseline, while reducing encoding and decoding complexity by about 84% and 13%, respectively.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Learned Image Compression, Complexity-Aware Design, Neural Architecture Search, Network Pruning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_AxpPOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies budget-constrained learned image compression through a unified lightweight variable-rate architecture. It first adjusts compute allocation between transforms and entropy modeling, then searches modules locally and composes them, and finally builds a model family through expansion and energy-based hierarchical pruning. Kodak and CLIC-PV results show favorable rate–distortion versus MAC trade-offs across several budgets. Experiments also report NVIDIA L20 timings and compare pruning with extended from-scratch training.

Critical strengths
  1. The study goes beyond a single compact codec and reports variable-rate models across a broad range of decoder budgets. For example, M1× improves compression substantially over Shallow-NTC while reducing encoder and decoder MACs. This makes the model family a useful empirical contribution to low-complexity compression.

  2. Comparisons with single-step pruning and extended from-scratch training help distinguish parameter-transfer benefits from simply spending more optimization effort. The observation that twice-longer from-scratch training approaches hierarchical-pruning performance is informative and appropriately narrows the interpretation to convergence efficiency.

Critical weaknesses
  1. The entropy/transform conclusion is derived mainly from 1/2/4-step variants near one 20-kMAC design, with transform width changing as entropy capacity changes. Local search then restricts each module to a narrow complexity range inherited from the base architecture, limiting opportunities to discover different allocations. Scaling and pruning that architecture do not by themselves show that the same procedure finds good allocations across budgets. Without matched search controls, the evidence supports a successful architecture recipe more strongly than a general method for deciding where compute should go.

  2. The motivation is constrained edge hardware, but timings are measured on a server GPU. Table 1 illustrates the limitation of the proxy: M1× has fewer decoder MACs than Shallow-NTC but takes 9.7 rather than 6.3 ms. Sequential entropy passes and operator efficiency can change the ranking. Consequently, favorable rate–distortion/MAC curves are insufficient to substantiate the targeted smartphone practicality, a central part of the paper's framing.

Questions for authors
  1. Does allocation plus local search outperform joint or random search at equal total training compute across multiple decoder budgets? How sensitive are the selected allocations to the initial architecture?

  2. Do the reported gains persist for complete codec latency on the intended smartphone or a constrained CPU, including entropy coding and data movement? How should the latency reversal against Shallow-NTC be interpreted?

2 comments
AuthorsOpenReview ↗

Response to Reviewer AxpP (1/2): Model-family design and cross-budget controls

Thank you for recognizing our model family's compression advantages across decoder budgets.

We have revised the manuscript, with changes in blue. Our contribution is a unified framework for constructing structurally compatible codec families across budgets. This response addresses allocation, search and initialization (Q1/W1); Part 2 presents complete smartphone measurements and the family's decoding benefits (Q2/W2).

Questions for Authors

Q1. Multiple budgets, training compute and initialization

Local search and hierarchical transfer serve model-family construction. The controls below assess allocation choices, search strategy and dependence on the starting architecture.

Allocation across budgets. Within each budget, encoder architecture is fixed and weights are independently trained. The added controls give the following BD-rates relative to the preselected 2-step references (Kodak/CLIC; Appendix E.2, Tables A6–A9):

BudgetTested allocationSeedsBD %, K/C
8k1-step, low entropy1−1.643/−4.216
20k1-step3, mean+3.472/+3.242
40k1-step, low entropy1+4.756/+4.756

The preferred tested allocation changes with budget. At 20k, 4-step has higher BD-rate than 2-step in all three seeds (mean +5.680/+5.759% relative to 2-step).

Search controls. Local and Random use the same candidate space and 20k-update candidate training, approximately matched search-process GPU time, and 1M final-training updates. Total training GPU time and actual MACs are not exactly matched. Random samples joint configurations; no other joint-search optimizer is tested.

The table summarizes three paired runs per budget. BD is Local relative to Random; negative favors Local. Cost change uses cumulative search-plus-final-training GPU time within each budget.

BudgetMean paired BD %, K/CObserved Local GPUh change
8k−0.288/−0.031−0.33%
20k+0.133/+0.388−7.51%
40k+2.315/+2.330+4.05%

The results do not establish a consistent RD advantage for Local. Neither strategy wins consistently at 8k/20k; Random wins all three 40k pairs. At 8k, paired Kodak BD ranges from −13.262% to +15.134%. Tables A1–A4 report all nine paired results, actual MACs and available cost breakdowns. The 8k/20k runs use A100; 40k uses H20.

Initialization. Changing the starting module configuration can improve or worsen final RD. Five completed runs from two alternative starting configurations, relative to original-start Local at the same seed, span −1.164% to +1.411% on Kodak and −1.690% to +1.271% on CLIC (Table A5).

Responses to Weaknesses

W1. Allocation scope and local candidate ranges

Sections 3.2–3.4 connect budget-guided allocation and module refinement to expansion that preserves block/channel correspondences for parameter transfer. The construction specifies both budget-specific architectures and their structural relationships.

The new allocation controls extend the analysis beyond the original 20k design. Local retains the base architecture's candidate ranges, and module composition need not improve monotonically. The search controls characterize this construction step within the tested settings.

The resulting family's RD and complete-decoding benefits are evaluated directly in revised Figure 4 and Part 2.

AuthorsOpenReview ↗

Response to Reviewer AxpP (2/2): Complete smartphone evaluation and decoding performance

Thank you for recognizing the compression gains across our decoder budgets. This second response addresses complete smartphone execution (Q2/W2); Part 1 covers model-family design and cross-budget controls.

We have revised the manuscript, with changes in blue. Figure 4 and Section 4.3 now present the family's measured resource and smartphone decoding performance.

Questions for Authors

Q2. Complete codec latency, entropy processing and data movement

The family's advantages extend to complete smartphone decoding, including CPU entropy processing and required tensor movement. x1/x2/x4/x8 improve native RD with fewer decoder MACs and parameters than Shallow-NTC/EVC-S/EVC-M/EVC-L, respectively, and decode faster on both tested phone images. x0.25 also improves RD and full codec latency over Shallow-NTC.

Paired values are Kodak/CLIC-PV (K/C). MAC/Param denotes decoder kMAC/pixel/M parameters. BD uses native actual-stream dataset RD relative to x1 over shared quality intervals; lower is better. Phone API times are medians.

ModelMAC/ParamBD %, K/CFull encode ms, K/CFull decode ms, K/C
Ours x0.254.94/1.10+15.96/+18.8440.5/256.834.8/227.7
Ours x117.94/2.510/045.8/363.342.9/286.5
Shallow-NTC20.51/10.47+19.93/+33.11205.3/1627.878.1/561.8
Ours x243.51/2.99−6.37/−7.5559.8/493.750.0/354.1
EVC-S69.56/10.92−1.22/+0.3484.3/709.772.5/626.8
Ours x481.34/6.25−9.54/−11.0990.6/754.866.4/501.9
EVC-M157.56/12.11−6.78/−5.67117.2/1395.181.4/693.3
Ours x8160.79/12.19−13.07/−14.97127.8/1056.675.4/620.0
EVC-L291.31/13.16−8.56/−7.61172.6/2203.5109.6/868.9

Tests use iPhone 17 Pro Max/CoreML ALL on kodim01 (512×768) and one CLIC image (1363×2048), H×W, at the model-specific operating points in Table A11. Complete APIs include CPU entropy coding/decoding, CDF/index/quantization, neural execution, required tensor movement and CPU-readable RGB8 output. Loading, compilation and disk I/O are excluded.

Appendix E.3 reports all configurations (Tables A12–A13), memory and explicit I/O (Table A14), and measured mean component shares (Figure 19). CoreML-internal movement remains included within CoreML/total time. Dataset RD and the two-image phone timings retain their respective scopes.

Responses to Weaknesses

W2. Practical deployment and the latency reversal

The original L20 neural times remain 9.7/6.3 ms for x1/Shallow-NTC, while x0.25 takes 3.5 ms. Sequential context execution and backend/operator efficiency affect latency beyond MACs. The L20 neural timings and complete phone APIs use different hardware, backends and timing scopes; these measurements do not isolate a single cause of the ranking change.

The new evidence directly tests the intended device and includes the entropy processing and data movement requested in Q2. Figure 4 jointly evaluates RD, parameters, computation, neural decoding and complete decoding, demonstrating the family's practical choices across budgets.

Reviewer_u1i1OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the learned image compression rate-distortion-complexity trade-off. This paper proposes a unified lightweight architecture and studies how computation should be allocated between the transforms and entropy model. Based on this analysis, the proposed method performs local structure search and module composition to identify efficient architectures. They further construct a scalable model family through gradual expansion and energy-based hierarchical pruning, which enables weight transfer across models of different complexities. Experiments on Kodak, CLIC-PV, and Tecnick demonstrate competitive rate-distortion performance across a range of complexity budgets.

Critical strengths
  1. The proposed method achieves good rate-distortion performance under computational constraints. The analysis of computation allocation between transforms and entropy models looks sound, and the observed bitrate-dependent trade-off provides insights for efficient codec design.

  2. The proposed method provides a new way to construct models across different complexity budgets. Rather than designing each model independently, this paper establishes structural correspondences across different model scales and employs hierarchical pruning to transfer weights from larger to smaller models. This is practically appealing, as it facilitates the design and optimization of a family of codecs with different complexity budgets.

Critical weaknesses
  1. DCVC-UF-Intra (Li et al., CVPR 2026) achieves strong rate-distortion performance with highly efficient encoding and decoding. Given that the paper claims favorable rate-distortion-complexity trade-offs across different budgets, the lack of comparison with DCVC-UF-Intra makes it difficult to fully assess the results of the performance. A comparison is needed.

  2. The paper primarily uses MACs/pixel to illustrate computational efficiency, but lower MACs/pixel does not necessarily translate into lower runtime latency, as already shown in the paper. For example, Table 1 shows that the proposed M2×1× model requires 17.9 kMACs/pixel for decoding, compared with 20.5 kMACs/pixel for Shallow-NTC. However, its actual decoding latency is 9.8 ms, which is higher than the 6.3 ms of Shallow-NTC.

  3. The paper motivates its design from the perspective of edge-device deployment; the latencies are measured on an NVIDIA L20 GPU. It's better to test on edge devices, and MACs/pixel may correspond to the real coding time on such devices.

  4. Several closely related works on rate-distortion-complexity (RDC) are not sufficiently discussed.
    Minnen and Johnston (ICIP 2023), Advancing the Rate-Distortion-Computation Frontier for Neural Image Compression, investigate the RDC frontier through architecture studies and construct a family of models with favorable trade-offs between computational cost and rate-distortion performance. Similarly, Zhang et al. (ICIP 2024), On Efficient Neural Network Architectures for Image Compression, systematically investigate transform architectures and context models to construct computationally efficient neural image codecs. These studies appear closely related to the motivation of this paper.

  5. The proposed unified architecture provides an innovative distinction from these previous studies. The proposed method, maintaining structural compatibility across model scales and transferring parameters through hierarchical pruning, can facilitate the construction of an efficient model family. This could be further strengthened to improve the paper.

  6. I think hierarchical pruning is a useful component of the unified framework, and it reduces training cost, but in most cases, we do not really care about the training budget. We should give a sufficient training budget since it does not impact the inference period. I am not sure about the actual contribution of this part.

Questions for authors

Please see the weaknesses for improvement. I would definitely reconsider the score if the issues could be resolved. Overall, this paper shows promising results.

6 comments
AuthorsOpenReview ↗

Response to Reviewer u1i1 (1/2): Comparisons and complete phone evaluation

Thank you for recognizing our compression performance and bitrate-dependent allocation insights. We have revised the manuscript, with changes in blue. This response addresses the requested baseline comparisons and complete smartphone evaluation (W1–W3). Part 2 addresses prior work, model-family construction and training evidence (W4–W6).

Questions for Authors

Our central contribution is a unified framework for constructing efficient codec families across budgets, combining transform–entropy allocation, module refinement, structure-preserving expansion and parameter transfer. Revised Figure 4 and Section 4.3 directly evaluate the resulting family's RD, decoder resources and smartphone decoding. The added comparisons below address the performance and deployment gaps you identified.

Responses to Weaknesses

W1. UF-Intra and the quality–resource comparison

DCVC-UF-Intra (UF-Intra) lies near the x8–x36 segment of our RD–MAC curve, while our family extends to much stricter budgets. The segment joins measured models; it does not represent an additional tested configuration at UF-Intra's budget.

K/C denotes Kodak/CLIC-PV. MAC/Param denotes decoder kMAC/pixel/M parameters. BD uses native actual-stream dataset RD, RGB8 PSNR and the common quality intervals of the 13 compared configurations, relative to x1; lower is better. Phone times are medians under the complete-API protocol in W3.

ModelMAC/ParamBD %, K/CFull encode ms, K/CFull decode ms, K/C
Ours x8160.79/12.19−13.07/−14.97127.8/1056.675.4/620.0
UF-Intra335.67/33.39−16.86/−16.75176.8/1434.7150.1/1227.0
Ours x36715.14/132.81−17.85/−20.03433.9/4229.9309.4/2391.6

x8 uses 52.1% fewer decoder MACs and 63.5% fewer decoder parameters than UF-Intra and completes phone decoding faster, with an RD trade-off. x36 improves RD on both datasets at greater cost. Figure 4 and Appendix E.3 show the full family and comparison set.

W2. MACs and the latency difference from Shallow-NTC

x0.25 is faster than Shallow-NTC in both the original L20 neural timing and complete phone decoding, with better native RD and fewer decoder MACs/parameters. The phone results also include x1 and UF-Intra:

ModelMAC/ParamBD %, K/CCoreML calls ms, K/CFull decode ms, K/C
Ours x0.254.94/1.10+15.96/+18.848.11/38.3834.76/227.65
Ours x117.94/2.510/015.06/80.1542.90/286.48
Shallow-NTC20.51/10.47+19.93/+33.1119.16/126.1178.05/561.76
UF-Intra335.67/33.39−16.86/−16.75104.96/816.48150.12/1227.01

Original L20 neural times at 1920×1088 are 3.5/9.7/6.3 ms for x0.25/x1/Shallow-NTC. Thus x1's original L20 latency trade-off remains explicitly reported. Sequential context execution and backend/operator efficiency affect latency beyond MACs; these measurements do not isolate their individual contributions. The CoreML column sums prediction calls inside complete decoding, rather than using the separate neural-only experiment.

W3. Evaluation on the intended device

Complete smartphone APIs include CPU entropy coding/decoding, CDF/index/quantization, neural execution, required tensor movement and CPU-readable RGB8 output. Loading, compilation and disk I/O are excluded.

Tests use iPhone 17 Pro Max/CoreML ALL on kodim01 (512×768) and one CLIC image (1363×2048), H×W, at the model-specific operating points in Table A11. Each direction has 90 formal calls across three processes, after ten warm-ups per process. The tables report median times. Dataset RD and these two-image phone measurements have distinct scopes.

Appendix E.3 provides all configurations (Tables A12–A13), reconstruction checks, memory and explicit I/O (Table A14). Figure 19 adds the measured complete-codec component breakdown. It reports mean component shares; CoreML-internal movement remains included within CoreML/total time without separate attribution.

The family therefore provides concrete choices across budgets: x0.25/x1 improve native RD and complete phone decoding over Shallow-NTC; x8 offers lower computation and faster phone decoding than UF-Intra, while x36 extends the family to better RD at higher cost.

AuthorsOpenReview ↗

Response to Reviewer u1i1 (2/2): Model-family design and training evidence

Thank you for highlighting the practical value of structural compatibility and parameter transfer across model sizes. This second response addresses W4–W6; Part 1 covers the requested baseline comparisons and complete phone measurements.

We have revised the manuscript, with changes in blue, to make the unified model-family construction and the evidence for each component clearer.

Questions for Authors

Our contribution is a unified framework for designing and constructing efficient codec families across budgets. Allocation and module refinement establish a compact base; expansion creates corresponding architectures; parameter transfer provides training routes between sizes. The resulting family's measured RD and decoding benefits are reported separately from the benefit of a particular pruning schedule.

Responses to Weaknesses

W4. Relationship to prior architecture studies

We now explicitly discuss Minnen and Johnston (2023), who study architecture choices and construct RD–computation model families, and Zhang et al. (2024), who investigate transform/context combinations and variable-rate codecs.

Our distinction is the combination of strict-budget allocation and structurally compatible family construction within one lightweight architecture. Sections 3.2–3.4 connect adjustable transform/entropy capacity, module refinement, expansion and parameter transfer. The family spans approximately 5–160 decoder kMAC/pixel, with x36 as a higher-budget extension.

The added fixed-encoder-architecture controls also examine allocation at 8k/20k/40k (Appendix E.2, Tables A6–A9). For example, the tested low-entropy 1-step allocation changes from −1.643%/−4.216% at 8k to +4.756%/+4.756% at 40k relative to the preselected 2-step references (Kodak/CLIC). These two grids use one seed; the 20k controls use three. This supports budget-dependent allocation choices in the tested design space.

W5. Strengthening the model-family contribution

Expansion preserves block/channel correspondences across model sizes, enabling parameter transfer through mapped blocks and consistently selected channels. The construction therefore specifies both the budget-specific architectures and their structural relationships (Section 3.4; Appendix B.2).

x1/x2/x4/x8 improve native RD with fewer decoder MACs and parameters than Shallow-NTC/EVC-S/EVC-M/EVC-L, respectively, and decode faster on both phone test images. Updated Figure 4 brings these comparisons into the main paper:

ModelDec. kMAC/pixelDec. params (M)BD %, K/CFull decode ms, K/C
Ours x117.942.510/042.9/286.5
Shallow-NTC20.5110.47+19.93/+33.1178.1/561.8
Ours x243.512.99−6.37/−7.5550.0/354.1
EVC-S69.5610.92−1.22/+0.3472.5/626.8
Ours x481.346.25−9.54/−11.0966.4/501.9
EVC-M157.5612.11−6.78/−5.6781.4/693.3
Ours x8160.7912.19−13.07/−14.9775.4/620.0
EVC-L291.3113.16−8.56/−7.61109.6/868.9

K/C denotes Kodak/CLIC-PV. BD uses native actual-stream dataset RD relative to x1 over shared intervals; time is complete-API median on one fixed phone input per dataset, including CPU entropy processing and tensor movement. Part 1 and Appendix E.3 give the protocol and all configurations.

W6. Hierarchical pruning and training effort

Hierarchical transfer trains multiple budget variants along the same structural path. The x8→x4→x2→x1 schedule produces trained intermediate models, so training effort concerns the family as well as the terminal x1.

Table 3 reports:

Reported routeKodak BD vs VTM-24.0, %
Single-step pruning+9.10
Equal training epochs+9.94
2× training epochs+8.94
Full pruning+8.87

Full pruning is 0.23 percentage points below single-step pruning. Extending training reduces the gap: the 2×-epoch result is 1.00 point below equal epochs and only 0.07 points above full pruning.

We also added a training-route comparison in Appendix E.4 (Table A15), with Direct480 as reference:

Training routeKodak BD vs Direct480, %
Direct4800
Hierarchy, 3×160 epochs+0.9118
Scratch640+0.0361

Direct480 has the lowest measured terminal BD-rate in this comparison, with Scratch640 within 0.04%. The terminal RD ranking varies across these training settings. Section 4.4 now explains this result alongside the intermediate variants produced by hierarchical transfer. Table A16 reports training updates, which are distinct from GPU time. These controls characterize training choices within the model-family design evaluated in Figure 4.

Thank you for the extensive new experiments and revised manuscript. Before updating my score, I would like clarification on:

  1. In E4, x1 is about 2× slower than Shallow-NTC on the phone neural path (20.4 vs. 10.2 ms at 1920×1088). In E5, however, Shallow-NTC’s CoreML time on the CLIC image is 126.1 ms versus 80.1 ms for x1, which reverses the ranking. What explains this discrepancy?

  2. Appendix E.4 states that the evaluated x1 comes from direct transfer. Please clarify which training route (hierarchical, direct, or scratch) was used for each of the other reported models.

Comments on the paper

2 comments
AuthorsOpenReview ↗

[General Response] Main Contributions & Outlook (1/2): Where Compute Goes and How Models Scale

We really appreciate the thoughtful feedback across all reviews, which led us to run additional tests on compute allocation, search, training strategies, and on-device execution. In this first comment, we summarize what our paper and these new experiments reveal about allocating compute and scaling into a model family.

1. Where should the compute go?

In learned compression, the transforms build the latent representation, while the entropy model codes it efficiently. Under a fixed decoding budget, spending more compute on one leaves less for the other.

Our original comparison around 20k MACs/pixel shows that this trade-off depends on bitrate: transform-heavy designs work better at high rates, while stronger context modeling helps at lower rates (Figure 3, left). Our newly added experiments show that this optimal balance also shifts with the computation budget:

Decoder budgetAlternative to the two-step referenceBD-rate %, Kodak / CLIC-PV
8kOne-step, low entropy capacity−1.643 / −4.216
20kOne-step+3.472 / +3.242
20kFour-step+5.680 / +5.759
40kOne-step, low entropy capacity+4.756 / +4.756

(Negative values favor the listed alternative. The encoder architecture is fixed within each budget. The 20k numbers are averaged over three seeds, while the 8k/40k controls use one seed; see Appendix E.2, Tables A6–A9.)

At 8k, the transform-heavy choice wins; at 40k, the two-step reference comes out ahead. At 20k, two-step outperforms both alternatives across all three seeds. Combined with our bitrate analysis, these results point to a practical design rule: lock in the right compute balance for your target budget and rate range first, then refine the architecture.

2. How can one good design serve different budgets?

Once we find a strong base design, we want to carry its architectural strengths across different model sizes. After improving the base via local module refinement (Figure 3, right), we expand the design while tracking block and channel correspondences across compatible sizes. These explicit mappings allow smaller models to directly inherit parameters from trained larger ones.

The resulting model family lets us quantify what additional compute actually buys:

Family memberDecoder kMACs/pixelNative BD-rate vs. ×1, %, Kodak / CLIC-PV
×117.940 / 0
×243.51−6.37 / −7.55
×481.34−9.54 / −11.09
×8160.79−13.07 / −14.97

(Results are evaluated on actual bitstreams; see Tables A12–A13. The ×0.25 and ×0.5 variants extend coverage to ultra-low budgets, while ×36 explores a much higher-compute regime.)

The core value of this family is that architecture, training, and performance are tightly coupled across scales: expansion defines structurally aligned architectures, parameter transfer reuses learned weights between compatible members, and the measured RD–compute curve maps out the exact compression performance achievable at each budget.

In the follow-up discussions, we also added Local/Random and starting-configuration controls, training-route comparisons, and the exact training path for each reported checkpoint to make the construction pipeline fully transparent and easy to verify.

What to examinePaper evidenceRelated discussion
Budget- and bitrate-dependent allocationSection 3.3; Figure 3, left; Tables A6–A9Reviewer AxpP (Q1); Reviewer u1i1 (W4)
Module refinement and compatible family constructionFigure 3, right; Section 3.4; Appendix B.2Reviewer u1i1 (W5)
Search choices and actual training routesAppendices E.1/E.4; Tables A1–A5 & A15–A17Reviewer AxpP (Q1); training-route follow-up
The family's measured RD–compute curveFigures 1 & 4; Tables 1–2 & A12–A13Summarized in Part 2

In Part 2, we will present the family's SOTA comparisons and on-device phone benchmarks, followed by the broader design takeaways we hope these findings encourage.

AuthorsOpenReview ↗

[General Response] Main Contributions & Outlook (2/2): Practical Speed & Future Directions

Our model family achieves state-of-the-art RD–complexity trade-offs across the evaluated low-complexity budgets of roughly 5–160 decoder kMACs/pixel. The follow-up discussions have strengthened our practical evidence and helped us articulate what this family-based perspective could mean for future codec design.

3. Do these models help in practice?

Across the family, ×1, ×2, ×4, and ×8 achieve lower native BD-rates with fewer decoder MACs and parameters than Shallow-NTC, EVC-S, EVC-M, and EVC-L, respectively, on both Kodak and CLIC-PV (Figure 4; Tables A12–A13).

For a concrete on-device benchmark, we focus on ×0.25 versus Shallow-NTC under FP16:

DatasetModelNative BD-rate vs. ×1, %FP16 neural decoding, msFP16 speedup vs. Shallow-NTC
KodakOurs ×0.25+15.9634.0221.17×
KodakShallow-NTC+19.9344.7171.00×
CLIC-PVOurs ×0.25+18.84412.5631.19×
CLIC-PVShallow-NTC+33.10814.9091.00×

(BD-rates are evaluated on actual bitstreams relative to ×1 and are shared across precision settings; Shallow-NTC's native RD includes online optimization. Latencies are 90-call medians from aligned FP16 neural replays on a fixed input per dataset using an iPhone 17 Pro Max / CoreML ALL; see Tables A12–A13, Table A14a, and Figure 20.)

×0.25 pairs stronger native RD and faster FP16 neural decoding with 75.9% fewer decoder MACs and 89.5% fewer decoder parameters.

We also benchmarked end-to-end coding—including CPU entropy processing, neural execution, and required tensor movement. Under the Mixed complete-codec policy, ×0.25 decodes in 34.8 / 227.7 ms, compared to 78.1 / 561.8 ms for Shallow-NTC on the fixed Kodak / CLIC-PV inputs, while also achieving faster encoding on both datasets (see Table A11 for operating points, and Tables A12–A13 and Figure 19 for full-codec latencies and component breakdowns).

Additional FP32 and Mixed neural replays confirm that ×0.25 remains faster than Shallow-NTC across all six tested image/precision combinations, while our stage-level profiling explains the precision-sensitive ordering between ×1 and Shallow-NTC (Appendix E.3.9).

At higher budgets, the added DCVC-UF-Intra comparison highlights the flexibility of our family: ×8 reduces decoder MACs by 52.1% and parameters by 63.5%, completing on-device decoding faster under the Mixed policy with a slight RD trade-off, whereas ×36 surpasses its native RD on both datasets at a higher compute budget.

4. Can we rethink what progress in codec design should look like?

Both learned image compression and hybrid coding continue to advance—from HEVC through VVC/VTM to ongoing ECM development, alongside previous generations of learned image and video codecs. We believe it is time to look beyond progress measured one standalone codec at a time and ask what a unified design methodology can offer across diverse devices, latency constraints, and application scenarios.

The goal is a better performance–compute frontier, offering strong, practical choices across the entire spectrum. A smartphone under a tight decoding budget and a high-end device with abundant compute should each have a well-matched model ready to deploy. As hardware evolves, the same framework provides a systematic way to explore new operating points.

This mirrors the value of an LLM family: practitioners can pick a model size tailored to their hardware constraints, while researchers can study how an architectural improvement scales across model capacities. For codecs, this curve makes the exact compute cost of higher compression efficiency explicit.

Our contribution in this direction is a concrete framework connecting budget-dependent allocation, structurally aligned architectures, cross-scale parameter transfer, and measured coding performance. We hope this encourages methods that bring stronger compression to today's devices while scaling naturally with tomorrow's hardware.

For readers looking for a quick visual summary, Figure 4 integrates the family's RD performance, decoder complexity, and end-to-end phone decoding latencies, while Figure 20 details the precision study. The related discussions can be found in Reviewer u1i1 (W1–W3, W5), Reviewer AxpP (Q2), and our precision follow-up.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.