acceptodds
Under review as a conference paper at ICLR 2027

ASRIC: Adaptive Sparse Routing for Learned Image Compression

Abstract

Learned image codecs often allocate transform computation and entropy-model capacity uniformly across heterogeneous content. As such, low-complexity regions incur the same dense processing as structurally complex regions, while a fixed pool of entropy-model features accommodates diverse conditional latent distributions. This allocation can incur redundant computation on unnecessary transform updates and limit the specialization of probability estimates to local context, resulting in limited compression performance under a fixed computational budget. In this paper, we propose Adaptive Sparse Routing for Learned Image Compression (ASRIC) to match transform computation and entropy-model capacity to content through two complementary mechanisms. Within the transforms, Spatially Adaptive Mamba blocks are introduced to select spatial tokens and scan directions for recurrent processing based on the Mamba module, while coordinate-dependent state transitions preserve the geometric separation between selected tokens. For entropy modeling, decoder-available context selects Top-K prototype banks, providing specialized statistical features for each prediction without transmitting bank indices. Experimentally, ASRIC achieves BD-Rate reductions of 19.15%, 18.67%, and 20.37% relative to VTM-9.1 on Kodak, CLIC, and Tecnick, respectively. Compared with CMIC, the strongest learned baseline in our comparison, ASRIC improves the VTM-referenced BD-Rate by 0.94, 0.81, and 0.32 percentage points across the three benchmarks, while requiring 4.2% fewer FLOPs and reducing encoding and decoding time by 5.6% and 16.3%, respectively.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Learned Image Compression, Entropy Model, Sparse Routing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_tfBiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ASRIC, a learned image codec with two content-adaptive mechanisms. In the transforms, Spatially Adaptive Mamba blocks run the selective scan only on a learned subset of tokens and scan directions, and scale the discretization step by the L1 grid distance between consecutive retained tokens. In the entropy model, a DCAE-style learned dictionary is split into 8 banks of 16,384 prototypes. A decoder-causal query selects the Top-2 banks per spatial position and channel group, and one retrieval is shared by the group's 80 channels, so no routing indices are transmitted. All learned baselines are retrained on a 500k-image ImageNet subset. ASRIC reports BD-rates of -19.15 / -18.67 / -20.37% vs VTM-9.1 on Kodak / CLIC / Tecnick (0.3-0.9 points better than CMIC), with 4.2% fewer FLOPs and lower decoding time on an RTX 4090. Ablations cover selector policy, gap rule, bank count, prototypes per bank, Top-K and the memory update rule.

Critical strengths
  1. Both routes are reproducible at the decoder without side information, and causality, CDF integerization and tie-breaking are specified carefully (A.4-A.5). This matters because adaptive entropy models easily break encoder/decoder determinism; here the design is described well enough to be implemented.

  2. The comparison is controlled: all learned baselines are retrained on the same data and timed through one wrapper on the same hardware, and the ablation set is broad. This is the main reason I take the relative trends seriously despite the concerns below.

Critical weaknesses
  1. The efficiency claims rest on numbers that look internally inconsistent. (a) Decoding time. Sec. 3.1 and A.4 describe a raster-order spatial context inside each group, i.e. a fully serial decode over positions. A ~2K CLIC image has about 11k latent positions at 1/16 resolution, so four groups give ~44k sequential steps. Each step includes routing, attention over 2 x 16,384 prototypes, the parameter head and arithmetic decoding of 80 symbols. A decode time of 0.41 s implies roughly 10 µs per step, about the cost of a single kernel launch. In the same table, Cheng2020-attn, which also uses a serial spatial context, takes more than 10 s. (b) FLOPs. A.4 gives about 70.87 GFLOPs per pass for Top-2 retrieval at 768x512, i.e. about 0.07 TFLOPs per extra bank for encoder plus decoder. The increments in Table 9 (+0.06 / +0.12 / +0.44 T for Top-1->2, Top-2->4, Top-2->Dense) match this almost exactly. But the protocol says FLOPs are CLIC native-resolution averages, where there are roughly 6-7x more latent positions, so dense retrieval should add about 3 T. The advantage over CMIC is only 0.10 TFLOPs (4.2%) and 0.08 s of decoding time, so if either accounting is off, the efficiency claim does not hold.

  2. The RD gains are small relative to plausible noise, and the contribution is incremental. The margin over CMIC is 0.94 / 0.81 / 0.32 points, and most ablation gaps are 0.05-0.4 points, apparently from single training runs. The two mechanisms share no decisions (Sec. 3.1), and each is a modest extension: token/direction pruning for vision SSMs, and MoE-style Top-K selection over a DCAE dictionary. Tables 7 and 9 show that more memory does not help (dense retrieval is only 0.05 points better than Top-2), so the "richer memory under a bounded budget" motivation is not really supported. The sparsity savings are also measured against an expensive starting point (full-scan ASRIC costs 5.31 TFLOPs) rather than against a dense model with the same FLOPs. Without seed variance and an iso-FLOP dense baseline, I cannot attribute the gains to adaptive routing.

Questions for authors
  1. Which decoding schedule produced the times in Tables 1 and 6: raster order as described, a wavefront, or something else? Are the reported rates computed from bitstreams decoded with that schedule, with bit-exact reconstruction?
  2. At which input resolution were the FLOPs in Tables 1, 2 and 9 computed? If the Top-K increments are at 768x512, please give corrected CLIC numbers.
  3. Which HPCM variant (Base or Large) was retrained, and what schedule did each baseline follow? In Table 1, HPCM looks weak (-16.29% on Kodak) compared with published HPCM-Large results; how do the official checkpoints compare?
  4. Can you report results over 2-3 seeds for ASRIC and CMIC, and a dense SAM baseline scaled to about 2.3 TFLOPs?
6 comments
AuthorsOpenReview ↗

Response to Reviewer tfBi

Thank you for your time and detailed feedback. Below, we provide detailed responses to your questions. We would appreciate hearing whether these responses address your concerns.

W1. Decoding time and FLOP accounting

W1.1. Decoding latency and execution

Thank you for examining the decoder. The reported decoder follows channel-group order and raster order within each group. Table R10 places the main time difference from Cheng2020-attn in primary-latent entropy decoding and host overhead.

Table R10: Decoding-stage times (s) of Cheng2020-attn and ASRIC on RTX 4090 and A100, using CLIC and batch size one. Strictly lower times within each platform are bold.

StageRTX 4090 / Cheng2020-attnRTX 4090 / ASRICA100 / Cheng2020-attnA100 / ASRIC
Hyperlatent decoding0.0080.0060.0070.005
Hyper-synthesis0.0030.0020.0020.002
Primary-latent entropy decoding10.4860.2958.9530.252
Synthesis transform0.0830.1050.0690.087
Other overhead3.0980.0032.8760.003
Total13.6780.41111.9070.349

Primary-latent entropy decoding includes context prediction, probability computation and entropy decoding. Other overhead covers host dispatch, transfers and synchronization waits outside those operations; non-overlapping intervals prevent double counting.

On RTX 4090, primary-latent entropy decoding takes 10.486 s for Cheng2020-attn and 0.295 s for ASRIC; other overhead takes 3.098 s and 0.003 s. On A100, these pairs are 8.953/0.252 s and 2.876/0.003 s. ASRIC's synthesis is slightly slower in this comparison, so synthesis does not explain the overall difference.

Cheng2020-attn inherits CompressAI's _decompress_ar [1]. At every raster position, its Python loop computes context and distribution parameters, converts scale indexes to a Python list, calls CPU RansDecoder.decode_stream, and copies decoded values into a tensor for the next position. With neural prediction on the GPU, this loop incurs repeated host dispatch, conversion and synchronization, whose costs accumulate across serial positions. The measured 3.098 s of other overhead on RTX 4090 is a substantial part of its total decoding time.

ASRIC reuses hyperprior and completed-group features. Each group–position pair performs one retrieval shared by 80 channel predictions; the linear readout projection follows prototype aggregation. This avoids repeating retrieval for every channel and limits the work added by the prototype memory. Table R11 gives the logical operator order. ASRIC still has 4HyWy4H_yW_y dependent steps, where Hy,WyH_y,W_y are latent dimensions. In the measured RTX 4090 implementation, the entropy stage and other overhead together fall from 13.584 to 0.298 s. These stages account for the large runtime difference. The shared readout reduces ASRIC's own retrieval cost; Cheng2020-attn has no prototype retrieval and already predicts its channels jointly.

Table R11: Operator order implied by the manuscript's causal decoder. Counts are logical invocations; fusion determines the number of device kernels.

StageOrdered operations and dependencyCount
InitializationDecode hyperlatents, run hyper-synthesis, prepare shared features.Once/image
Group contextApply the 1 × 1 projection to decoded groups; use zero channel context for the first group.Once/group
Spatial contextLocal masked 5 × 5 convolution; concatenate hyperprior, spatial and group features into a 320-dimensional context.QQ
Bank routingScore eight keys with a 64-dimensional query; select Top-2 and normalize their weights.QQ
Prototype retrievalProject the query to 88 dimensions; compute selected-prototype dot products, within-bank softmax and aggregation; project once to 192 dimensions.QQ
Distribution and decodingPredict 80 × 5 × 3=1,200 parameters, construct channel-specific integer CDFs, decode channels in order and update the current vector.QQ
ReconstructionAfter all group–position steps, run sparse synthesis and crop padding.Once/image

Here Q=GHyWyQ=GH_yW_y, with G=4G=4 channel groups and latent dimensions Hy,WyH_y,W_y; there are 80Q80Q primary-latent symbols. The four stages from spatial context to distribution decoding run in order at each group–position pair, updating the vector before the next position. CDF denotes cumulative distribution function.

Both devices use ten warm-ups and 50 timed repetitions per CLIC image, synchronization at call boundaries and equal-weight image averaging. Rates count the complete bitstream, and timing covers its full decoding path. Bit-exact verification compares recovered integer latents, bank indices and integer CDFs step by step, then compares reconstruction under the same numerical backend.

AuthorsOpenReview ↗

Response to Reviewer tfBi

W1.2. Input resolution and retrieval FLOPs

Thank you for identifying this discrepancy and helping us improve the manuscript. Tables 1 and 2 report FLOPs on CLIC Professional Validation. Its mean image width and height are approximately 1789 and 1189 pixels. Table R12 reports the updated CLIC counts alongside Kodak (768×512768\times512), with units stated explicitly.

Table R12: Capacity and retrieval ablations from Table 9. Kodak and CLIC Professional Validation FLOPs are reported in G and T, respectively; Kodak BD-Rate is relative to VTM-9.1.

ControlVariantTotal prototypesKodak FLOPs (G)CLIC FLOPs (T)Kodak BD-Rate (%)
Capacity/KB4,096 prototypes32,768280.521.74-18.31
8,192 prototypes65,536315.951.92-18.78
16,384 prototypes (Ours)131,072386.822.29-19.15
32,768 prototypes262,144528.553.03-19.12
RetrievalTop-1 (12.5%)131,072315.951.92-18.72
Top-2 (25.0%) (Ours)131,072386.822.29-19.15
Top-4 (50.0%)131,072528.553.03-19.18
Dense (100%)131,072812.024.51-19.20

All configurations store eight banks. Capacity/KB fixes Top-2; Retrieval fixes 16,384 prototypes per bank, with percentages indicating the fraction of active banks. Both Ours rows describe the same default model. Non-default Kodak counts are calculated from 386.82 G by adjusting prototype-attention FLOPs using the equation below and holding other operator counts fixed. Bold indicates the best BD-Rate within each ablation group.

The default Kodak count is 386.82 GFLOPs, or 0.38682 TFLOPs. Following A.4, the other Kodak entries vary the prototype dot-product and aggregation costs while holding the remaining operator counts fixed:

\frac{8Qd_m\bigl(KM-2\times16{,}384\bigr)}{10^9} \quad\text{GFLOPs},$$ where $F_{\mathrm{Kodak}}$ is the encoder-plus-decoder count, $K$ is the number of selected banks, $M$ is the number of prototypes per bank, $d_m=88$ is the prototype dimension, and $Q=4\times48\times32=6{,}144$ is the number of queries per coding pass. The factor eight counts dot products and aggregation in both encoding and decoding, with a multiplication and an addition counted separately. Thus, A.4's 70.87 GFLOPs is the Top-2 attention cost for one Kodak pass; reading one additional default bank in both passes also adds 70.87 GFLOPs. Note that the FLOPs reported in Table 9 of the original manuscript differ from those in Table R12, as the CLIC estimates used the Kodak per-bank increment. This underestimated the additional computation of reading more prototypes on CLIC. The updated counts retain the default configuration as the reference and use the increment for the corresponding dataset. CLIC counts average all 41 images after padding each image to multiples of 64; the mean dimensions above summarize the dataset rather than define one fixed input size. The conclusions of Tables 9 and R12 remain the same. At fixed Top-2 retrieval, 16,384 prototypes per bank give the best Kodak BD-Rate among the tested capacities. Increasing to 32,768 adds computation without improving RD. At fixed stored capacity, Top-2 achieves $-19.15\\%$ versus Dense's $-19.20\\%$, while reducing CLIC FLOPs from 4.51 to 2.29 T (49.2%). The main-table comparison also remains 2.29 TFLOPs for ASRIC versus 2.39 TFLOPs for CMIC. The RD scores and measured runtimes are unchanged.
AuthorsOpenReview ↗

Response to Reviewer tfBi

W2. Stability and the contribution of adaptive allocation

W2.1. Methodological novelty and separate routing decisions

Thank you for raising this concern. CMIC adapts scan order but still updates every token; DCAE adapts attention weights but still reads the full dictionary at each query. Thus, long-range updates also run where local features suffice, and enlarging the dictionary raises retrieval cost at every location. Our motivation is to make this computation selective: SAM allocates long-range updates to content that benefits from them, while MASKR gives each context access to a larger memory under a fixed read budget. This requires preserving local reconstruction features, spatial geometry and decoder-causal probability prediction.

SAM versus token pruning. Token pruning typically scores tokens and removes a subset from later processing. SAM instead keeps dense local features and routes only the long-range branch. It adapts both token count and scan directions under average compute constraints. Packing selected tokens also changes their spacing, so SAM uses L1 displacement between retained coordinates in the state transition. Pruning-aware state alignment is relevant prior work; our geometric interval differs from scan-rank intervals at row or column boundaries. The design therefore controls which recurrent updates execute while retaining the spatial information needed for reconstruction.

MASKR versus DCAE and MoE. MASKR selects banks before prototype attention, so stored capacity can grow without reading every prototype at each query. Conventional MoE selects separate neural experts; MASKR selects prototype features for a shared probability predictor. One readout serves 80 channels with distinct likelihood parameters. Selection uses only the hyperprior and previously decoded latents, allowing both codec ends to reproduce the coding CDFs without transmitting bank indices. The contribution is selective, shared access to predictive memory within a causal entropy model.

Why the decisions are separate. A region can benefit from long-range transform processing yet be predictable from decoded context. Its token score therefore does not determine which prototype banks are useful. The selectors use transform features and decoder-available context, respectively. They are coupled through the coded latents and RD loss: SAM changes what KAEM predicts, and KAEM's code-length estimate supplies the rate term that trains SAM. Separate decisions let each stage allocate its own resource using the information available to it.

W2.2. Independent training runs

Thank you for asking about stability. We report three independent training runs for ASRIC and CMIC, each covering all five RD points. ASRIC has better BD-Rate in every run on all three datasets, with Tecnick improvements of 0.32, 0.35 and 0.44 percentage points. The training subset, validation-based checkpoint selection and each method's training settings are fixed across runs.

Table R13: BD-Rate over three independent training runs (%, against VTM-9.1). The better method is bold within each run and in the mean comparison.

MethodRun / statisticKodakCLICTecnick
CMIC1-18.21-17.86-20.05
ASRIC1-19.15-18.67-20.37
CMIC2-18.27-17.94-20.12
ASRIC2-19.02-18.55-20.47
CMIC3-18.13-17.79-19.97
ASRIC3-19.24-18.73-20.41
CMICMean ± SD-18.203 ± 0.070-17.863 ± 0.075-20.047 ± 0.075
ASRICMean ± SD-19.137 ± 0.111-18.650 ± 0.092-20.417 ± 0.050
Mean improvement (points)0.9330.7870.370

Sample standard deviations use denominator n−1n-1, where n=3n=3 is the number of independent runs.

ASRIC's between-run standard deviations are 0.111, 0.092 and 0.050 points on Kodak, CLIC and Tecnick; its mean improvements are 0.933, 0.787 and 0.370 points. The runs show stable training and a consistent improvement under these settings.

Reviewer_q1oROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ASRIC, a learned image codec that adapts computational allocation in both the transforms and the entropy model. Experiments on Kodak, CLIC, and Tecnick report improved RD performance over the evaluated baselines, together with lower FLOPs and decoding latency than CMIC.

Critical strengths
  1. The paper investigates token and direction selection and coordinate-dependent transitions. Entropy routing uses only decoder-available information, and the shared retrieval still produces channel-specific likelihoods. These might address practical constraints of entropy decoding.

  2. The paper evaluates three standard datasets, compares against relevant recent codecs, and reports end-to-end timing on RTX 4090 and A100.

Critical weaknesses
  1. Although the paper reports FLOPs and runtime, it does not compare total model size or peak memory usage against the baselines. Consequently, the storage and memory costs relative to the computational savings remain unclear, limiting the assessment of practical efficiency.

  2. The incremental gains from several key design choices are modest. Dynamic thresholding, coordinate-aware intervals, and MASKR improve the Kodak BD-Rate score by only 0.32, 0.39, and 0.64 % over fixed Top-K token selection, unit intervals, and an MLP replacement, respectively (Tables 8 and 10).

Questions for authors
  1. What exactly replaces ASRM in Table 3? The MASKR-only configuration obtains −18.02% on Kodak, whereas “Full scan” in Table 4 obtains −19.32%. Please identify the architectural, transition-rule, and training differences between these configurations so that the effect of sparse routing can be interpreted correctly.

  2. Discussion on sequential entropy decoding versus the reported speedup. ASRIC’s spatially autoregressive schedule limits parallelism across latent positions, whereas CMIC uses a parallel SCCTX-based entropy model. Does ASRIC’s entropy-decoding stage itself take longer than CMIC’s, with the overall speedup coming from a faster synthesis transform, or is entropy decoding also faster?

  3. Within the same transform, causal context, and probability head, how does learned Top-2 routing compare with fixed or random two-bank selection at identical total memory capacity, and with a smaller dense dictionary matched to the active retrieval budget? These comparisons would clarify whether the gains depend on content-adaptive bank selection.

5 comments
AuthorsOpenReview ↗

Response to Reviewer q1oR

Thank you for your time and detailed feedback. Below, we provide detailed responses to your questions. We would appreciate hearing whether these responses address your concerns.

W1. Model storage and peak memory

Thank you for highlighting these costs. ASRIC uses 70.47 M active parameters and 4.7 GB peak memory, close to CMIC's 69.11 M and 4.5 GB. It improves Kodak/CLIC/Tecnick BD-Rate by 0.94/0.81/0.32 percentage points. Encoding takes 0.67 versus 0.71 s, and decoding takes 0.41 versus 0.49 s. Table R6 reports model size, memory, computation and RD together. ASRIC stores 172.56 M parameters; total and active parameter counts are equal for the other baselines.

Table R6: Resources, RD and runtime of learned codecs, in the order of Table 1. BD-Rate (%) is relative to VTM-9.1. FLOPs, memory and time are evaluated on CLIC; times use RTX 4090.

(a) Parameters, memory and computation

MethodTotal parameters (M)Active parameters (M)Peak memory (GB)FLOPs (T)
Cheng2020-attn26.6026.603.22.03
TCM-L75.8975.897.73.74
FTIC69.7869.785.02.38
HPCM-Base68.5068.505.92.00
DCAE119.22119.225.62.36
GLIC67.2067.205.52.48
MambaIC157.09157.0920.35.56
CMIC69.1169.114.52.39
ASRIC (Ours)172.5670.474.72.29

Table R6: RD and coding time of learned codecs (continued). BD-Rate (%) is relative to VTM-9.1; times are measured on CLIC with RTX 4090.

(b) RD and coding time

MethodEnc. (s)Dec. (s)KodakCLICTecnick
Cheng2020-attn5.31>10+4.46+5.92+3.97
TCM-L0.540.49-13.47-9.16-11.01
FTIC>10>10-14.69-10.54-14.27
HPCM-Base0.460.45-16.29-15.18-17.42
DCAE0.510.49-17.72-17.49-18.93
GLIC0.530.36-18.14-17.53-19.17
MambaIC1.170.58-15.28-15.87-15.91
CMIC0.710.49-18.21-17.86-20.05
ASRIC (Ours)0.670.41-19.15-18.67-20.37

M denotes millions. Total parameters cover the full stored model; active parameters participate in computation. Memory is the runtime peak. Times retain Table 1's two-decimal precision and >10>10 notation. Bold indicates the best BD-Rate on each dataset.

Our motivation is to make more model capacity available across different content while limiting the computation used for each prediction. The stored memory contains diverse statistical features, but each query reads only selected banks. SAM similarly allocates long-range updates to selected tokens and directions. ASRIC therefore achieves better RD than CMIC with similar active parameters and peak memory, fewer FLOPs and shorter coding times on RTX 4090. These results suggest that large sparse models are a promising way to improve image compression without a proportional increase in computation or runtime. The larger stored model remains a storage cost.

AuthorsOpenReview ↗

Response to Reviewer q1oR

W2. Interpretation of the individual design gains

Thank you for asking about the size of these gains. Each cited ablation changes one component while retaining the other proposed components, so it measures an additional gain within an already strong codec.

In Table 8, Fixed Top-KtK_t uses learned token scores but keeps a fixed number KtK_t of tokens. The 0.32-point gain therefore comes from adapting the retained count to content. Relative to Random selection, Uniform spatial grid and Feature-magnitude ranking, the gains are 1.47, 1.13 and 0.68 points at 2.27–2.34 TFLOPs. The 0.39-point interval gain shows that retaining geometric spacing helps after token removal. In Table 10, MASKR improves over the MLP replacement by 0.64 points for 2.29 versus 2.27 TFLOPs. These improvements come from choosing where to compute, preserving geometry during sparse propagation and adapting probability prediction to context.

We also train a joint configuration with Fixed Top-KtK_t, unit intervals and the MLP replacement, keeping the latent interface, causal context, direction-selection rule and other layers unchanged. It uses the same training data and five RD points. We set KtK_t on validation data to match ASRIC's average token budget and disable the unused token-budget and memory losses.

Table R7: Joint comparison of the three design choices. BD-Rate is relative to VTM-9.1; FLOPs and coding times use the CLIC evaluation protocol.

ConfigurationFLOPs (T)Enc. (s)Dec. (s)Kodak BD-Rate (%)CLIC BD-Rate (%)Tecnick BD-Rate (%)
Fixed Top-KtK_t + Unit + MLP2.260.720.40-18.63-18.07-19.69
ASRIC (Ours)2.290.670.41-19.15-18.67-20.37

The joint configuration uses a fixed token count, unit geometric intervals and an MLP. ASRIC uses dynamic thresholding, L1 intervals and MASKR. Bold indicates the best BD-Rate on each dataset.

Table R7 shows that simpler replacements save only 0.03 TFLOPs and 0.01 s in decoding, but raise BD-Rate by 0.52, 0.60 and 0.68 points on Kodak, CLIC and Tecnick and take 0.05 s longer to encode. The single-component gains should therefore be judged alongside their cost: these content-adaptive designs improve an already strong codec across datasets with little additional computation.

Q1. MASKR-only versus Full scan

Thank you for asking. Table 3's MASKR-only replaces ASRM with original SS2D and its native recurrence. Table 4's Full scan keeps adaptive token selection and coordinate-dependent transitions, but activates all four scan directions. The scores −18.02-18.02\\% and −19.32-19.32\\% therefore belong to different transform modules.

Both retain MASKR and the latent interface and are trained separately with the same data, RD points and training budget. Full scan retains the token-budget loss and disables the direction-budget loss. Within SAM, comparing Full scan with ASRIC (−19.32-19.32\\% versus −19.15-19.15\\%) isolates the 0.17-point RD cost of direction selection.

Q2. Entropy decoding versus synthesis time

Thank you for raising this question. ASRIC's entropy-decoding stage is slightly slower than CMIC's; the overall saving comes from the synthesis transform. ASRIC uses group-wise raster decoding and shares one retrieval across 80 channel predictions, while CMIC uses SCCTX.

Table R8: Decoding-stage times (s) on RTX 4090 and A100, using CLIC images and batch size one. Strictly lower times within each platform are bold.

StageRTX 4090 / CMICRTX 4090 / ASRICA100 / CMICA100 / ASRIC
Hyperlatent decoding0.0060.0060.0050.005
Hyper-synthesis0.0020.0020.0020.002
Primary-latent entropy decoding0.2790.2950.2390.252
Synthesis transform0.1990.1050.1690.087
Other overhead0.0020.0030.0020.003
Total0.4880.4110.4170.349

On RTX 4090/A100, the entropy stage adds 0.016/0.013 s, while synthesis saves 0.094/0.082 s. Including the other stages, complete decoding falls from 0.488/0.417 s to 0.411/0.349 s. SAM reduces synthesis work by scanning packed selected tokens only in active directions. The entropy-stage timings include context prediction, routing, attention, CDF construction and arithmetic decoding under the same timing boundaries.

The speed benefit depends on the device and coding direction: Table 6 reports A100 encoding times of 0.75 s for ASRIC and 0.64 s for CMIC.

AuthorsOpenReview ↗

Response to Reviewer q1oR

Q3. Bank selection and a smaller dense dictionary

Thank you for suggesting this ablation. At the same stored capacity and read budget, Learned Top-2 (Ours) improves Kodak BD-Rate by 0.55 and 0.50 percentage points over Fixed two-bank and Random two-bank, respectively (Table R9).

The six configurations use the same transform, causal context, probability head, retrieval dimensions and training schedule. Bank load balancing is disabled throughout this ablation; transform-budget losses are retained. The first four configurations store eight banks of 16,384 prototypes and read two banks per query: Learned Top-2 (Ours) selects banks from the causal context; Fixed two-bank always reads the same pair; Random two-bank uses a reproducible random pair determined by group and position; Balanced schedule cycles through all 28 pairs independently of content. Small dense stores one dictionary of 32,768 prototypes and reads all of them with a single attention normalization. Two-bank dense stores two banks of 16,384 prototypes and reads both, retaining separate within-bank attention and query-dependent bank weights. All six read 32,768 prototypes per query and are reproducible at the decoder.

Table R9: Bank-selection ablation. Stored and Read count stored prototypes and prototypes read per query; Content-based indicates whether bank selection depends on content. Kodak BD-Rate is relative to VTM-9.1; FLOPs and decoding time are averages on CLIC.

SelectionStoredReadContent-basedFLOPs (T)Dec. (s)Kodak BD-Rate (%)
Learned Top-2 (Ours)131,07232,768✓2.290.41-19.15
Fixed two-bank131,07232,768×2.280.40-18.60
Random two-bank131,07232,768×2.290.41-18.65
Balanced schedule131,07232,768×2.290.41-18.73
Small dense32,76832,768×2.280.40-18.56
Two-bank dense32,76832,768×2.280.40-18.41

Learned Top-2 improves over Balanced schedule by 0.42 points even though both access all eight banks across queries. Thus, matching banks to content adds value beyond distributing access across the memory. Its gains over Small dense and Two-bank dense are 0.59 and 0.74 points. These comparisons show the benefit of selecting relevant features from a larger memory while keeping the read budget fixed.


Thank you again for your time and effort. We believe that these experimental results will substantially strengthen the case for our paper, and we respectfully ask you to reconsider your overall assessment.

Reviewer_6mLKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ASRIC, a learned image codec that uses content-adaptive sparse routing in both the transforms and the entropy model. The transform selects spatial tokens and scan directions for Mamba processing, while coordinate-dependent state transitions account for the spacing between retained tokens. The entropy model uses decoder-available context to select Top-K prototype banks, avoiding the need to transmit routing indices. Experiments on Kodak, CLIC, and Tecnick report modest RD improvements over CMIC with lower computational cost. Ablation studies examine token selection, geometric intervals, memory capacity, and retrieval budgets.

Critical strengths
  1. The paper addresses a practical problem: allocating computation and entropy-model capacity according to image content. The two proposed mechanisms fit this motivation well. Spatial routing reduces redundant long-range processing, while bank routing allows access to a larger prototype memory under a limited retrieval budget. The entropy router uses the hyperprior and previously decoded latents, making its decisions reproducible at the decoder without additional routing information under the stated formulation.
  2. The evaluation provides useful evidence about the RD–complexity trade-off. In addition to comparisons on three datasets, the paper includes ablations of selection policies, scan directions, and retrieval density. I appreciate that the authors distinguish FLOP reductions from actual latency improvements and report hardware-dependent behavior, including cases where encoding becomes slower. The computational savings relative to full scanning, obtained at a small RD cost, are particularly encouraging.
Critical weaknesses
  1. The experiments do not sufficiently isolate the contribution of content-adaptive routing. Given the close relationship to dictionary-based entropy models and token pruning in SSMs, this distinction matters for assessing the contribution. Table 9 shows that Top-2 retrieval can approximate dense retrieval at a lower computational cost, but does not establish that learned bank selection outperforms random or fixed selection under the same budget. Comparisons with DCAE also involve changes to the transforms, context model, and predictor. Likewise, the L1 geometric interval is mainly compared with unit intervals, without a scan-rank interval control that would connect more directly to prior pruning-aware methods. I therefore find it difficult to determine how much of the improvement comes from routing itself rather than memory capacity or other architectural choices.
  2. Important training and implementation details remain unclear. The prototype-update target is defined as t_u = sg(W_m c_u), but the learning mechanism for W_m is not specified. The paper also does not fully describe how the RD loss propagates through hard token and direction selection to train the selectors. On the execution side, entropy decoding requires sequential context prediction and prototype retrieval at each spatial position, yet the implementation supporting the reported latency is insufficiently documented. Since both the modest RD gains and the practical speed improvements are central to the paper, these gaps limit my confidence in the reproducibility of the results.
Questions for authors
  1. With the transforms, context model, probability head, total prototype count, and retrieval budget held fixed, how much does learned Top-2 bank selection improve over random or fixed bank selection? Do any existing ablations isolate this comparison?
  2. How is W_m initialized and updated? If it is used only within sg(W_m c_u), where does its learning signal come from? Please also clarify the forward computation and backward surrogate for hard token and direction selection, including how the RD loss reaches the selectors through the gathered scan outputs.
  3. The improvement over CMIC on Tecnick is 0.32 percentage points. Are results from independent training runs or per-image measurements available to assess its stability? Please also clarify the BD-Rate integration interval, interpolation and aggregation procedures, and the VTM color format and PSNR evaluation domain.
  4. According to the normalization in Eq. (32), an individual bank’s activation ratio cannot exceed 0.5 when K=2. However, some cells in Fig. 1(b) appear close to 1.0 on the color scale. Was an additional normalization applied? Please clarify the plotted statistic or provide the underlying numerical matrices.
5 comments
AuthorsOpenReview ↗

Response to Reviewer 6mLK

Thank you for your time and detailed feedback. Below, we provide detailed responses to your questions. We would appreciate hearing whether these responses address your concerns.

W1. Contribution of content-adaptive routing

W1.1. Bank-selection ablation at matched capacity and retrieval budget

Thank you for highlighting this distinction. ASRIC and DCAE differ in their transforms, context models and probability heads. Their overall performance difference therefore combines several architectural changes. Table R1(a) compares the complete codecs. ASRIC improves Kodak/CLIC/Tecnick BD-Rate by 1.43/1.18/1.44 percentage points, with 2.29 versus 2.36 TFLOPs. Its decoding is faster, while encoding takes longer. Table R1(b) isolates bank selection within ASRIC.

The six configurations use the same transform, causal context, probability head, retrieval dimensions and training schedule. Bank load balancing is disabled throughout this ablation; transform-budget losses are retained. The first four configurations store eight banks of 16,384 prototypes and read two banks per query: Learned Top-2 (Ours) selects banks from the causal context; Fixed two-bank always reads the same pair; Random two-bank uses a reproducible random pair determined by group and position; Balanced schedule cycles through all 28 pairs independently of content. Small dense stores one dictionary of 32,768 prototypes and reads all of them with a single attention normalization. Two-bank dense stores two banks of 16,384 prototypes and reads both, retaining separate within-bank attention and query-dependent bank weights. All six read 32,768 prototypes per query and are reproducible at the decoder.

Table R1: (a) Complete-codec comparison of DCAE and ASRIC; (b) bank-selection ablation within ASRIC. BD-Rate (%) is relative to VTM-9.1; FLOPs and times are averages on CLIC.

(a) Complete codecs

MethodFLOPs (T)Enc. (s)Dec. (s)KodakCLICTecnick
DCAE2.360.510.49-17.72-17.49-18.93
ASRIC (Ours)2.290.670.41-19.15-18.67-20.37

(b) Bank-selection ablation

SelectionStoredReadContent-basedFLOPs (T)Dec. (s)Kodak BD-Rate (%)
Learned Top-2 (Ours)131,07232,768✓2.290.41-19.15
Fixed two-bank131,07232,768×2.280.40-18.60
Random two-bank131,07232,768×2.290.41-18.65
Balanced schedule131,07232,768×2.290.41-18.73
Small dense32,76832,768×2.280.40-18.56
Two-bank dense32,76832,768×2.280.40-18.41

Stored and Read count stored prototypes and prototypes read per query. Content-based indicates whether bank selection depends on content. Bold marks the best BD-Rate on each dataset within each panel.

At the same stored capacity and read budget, Learned Top-2 (Ours) improves over Fixed two-bank, Random two-bank and Balanced schedule by 0.55, 0.50 and 0.42 points. Balanced schedule already accesses all eight banks across queries, so matching banks to content adds value beyond distributing access. Learned Top-2 also improves over Small dense and Two-bank dense by 0.59 and 0.74 points at the same read budget. These comparisons identify the benefit of selecting relevant features from a larger memory.

AuthorsOpenReview ↗

Response to Reviewer 6mLK

W1.2. Scan-rank intervals and boundary handling

Thank you for highlighting this distinction. L1 measures displacement on the feature grid; scan-rank difference measures distance along the scan order. Our ablations compare both rules and line-boundary handling. For a width-eight raster scan, (0,7)→(1,0)(0,7)\to(1,0) has L1 distance eight and rank difference one; (0,1)→(1,1)(0,1)\to(1,1) has distances one and eight.

Carry propagates state across lines; Reset starts each line with zero state and sets the first retained token's interval to one. L1 and scan-rank intervals coincide within a line, so their Reset variants share an implementation and checkpoint.

Table R2 replays identical weights, token masks and directions, changing only the recurrence. It isolates the immediate effect of each rule. We average relative changes in Lℓ=Rℓ+λℓDℓ\mathcal L_\ell=R_\ell+\lambda_\ell D_\ell across the five RD points, using L1–Carry (Ours) as the reference at each point. Here ℓ\ell indexes RD points, RℓR_\ell is estimated rate, DℓD_\ell is distortion and λℓ\lambda_\ell is its weight.

Table R2: Interval ablation with fixed weights and identical routing. Loss is normalized to L1–Carry.

IntervalBoundaryKeep ratioMean directionsΔLRD\Delta\mathcal L_{\rm RD} (%)
UnitCarry0.5042.01+0.74
UnitReset0.5042.01+1.05
Scan-rankCarry0.5042.01+0.31
Scan-rankReset0.5042.01+0.46
L1 (Ours)Carry0.5042.010.00
L1Reset0.5042.01+0.46

Table R3 retrains each configuration under the same average budgets. The masks remain image-dependent. Similar keep ratios and direction counts explain the close FLOPs and times; runtime is rounded to 0.01 s.

Table R3: Trained interval and boundary variants. Kodak reports BD-Rate against VTM-9.1; computation and time are averages on CLIC.

IntervalBoundaryKeep ratioMean directionsFLOPs (T)Enc. (s)Dec. (s)Kodak BD-Rate (%)
UnitCarry0.5022.012.270.660.40-18.76
UnitReset0.5012.002.280.660.40-18.62
Scan-rankCarry0.5052.012.300.670.41-18.98
Scan-rankReset0.5042.012.290.670.41-18.94
L1 (Ours)Carry0.5042.012.290.670.41-19.15
L1Reset0.5042.012.290.670.41-18.94

With fixed routing, unit intervals, scan-rank intervals and line resets increase RD loss by 0.74%, 0.31% and 0.46%. After retraining, L1–Carry retains Kodak BD-Rate advantages of 0.39, 0.17 and 0.21 points at 2.27–2.30 TFLOPs. Thus, accounting for grid displacement and carrying state across lines improves coding efficiency even after each model adapts to its transition rule.

W2. Training and decoding implementation

W2.1. Prototype-target projection

Thank you for requesting this clarification. WmW_m is not learned and remains fixed after initialization. We sample a random initialization matrix A∈R320×88A\in\mathbb R^{320\times88} with independent standard Gaussian entries and apply a thin QR decomposition:

A&=QR,\qquad W_m=Q^\top,\\\\ t_u&=\operatorname{sg}(W_m c_u)=\operatorname{sg}(Q^\top c_u), \end{aligned}$$ where $Q\in\mathbb R^{320\times88}$ has orthonormal columns and $R\in\mathbb R^{88\times88}$ is upper triangular. For each image–group–position query $u$, $c_u\in\mathbb R^{320}$ is the causal context and $t_u\in\mathbb R^{88}$ is the prototype-update target; $\operatorname{sg}$ denotes stop-gradient. The resulting $W_m$ has orthonormal rows and is stored as a fixed buffer outside the optimizer. Keeping $W_m$ fixed prevents EMA from mixing targets computed using different projections, while the context features remain trainable. The resulting $t_u$ serves as the target for EMA updates of the assigned prototype values.
AuthorsOpenReview ↗

Response to Reviewer 6mLK

W2.2. Hard selection and RD gradients

Thank you for this important question. Hard selection determines the forward computation; straight-through gates connect the RD loss to the selectors:

\qquad \widetilde r_d=\operatorname{sg}(r_d^h-\pi_d)+\pi_d,$$ where $n$ indexes tokens, $d$ indexes directions, $p_n,\pi_d$ are selection probabilities, and $m_n^h,r_d^h$ are safeguarded binary decisions. The gates equal the hard decisions in the forward pass and use probability derivatives in the backward pass. Selected features are gathered, scanned and scattered to form directional outputs $U_d$, with zero outside the selected support. Training evaluates four directions on that support, with the hard gates selecting the forward contribution: $$Z=\frac{\sum_d\widetilde r_dU_d}{\sum_d\widetilde r_d},\qquad Y_n=(1-\widetilde m_n)V_n+\widetilde m_n Z_n,$$ where $V_n$ is the pre-scan feature, $Z$ is the fused scan output and $Y_n$ is the gated feature. The nonempty-direction safeguard keeps the denominator positive. Inference executes only active directions. Holding scan outputs fixed, the direct gate derivatives are: $$\left.\frac{\partial Y_n}{\partial p_n}\right|_{\rm gate}=Z_n-V_n, \qquad \left.\frac{\partial Z}{\partial\pi_d}\right|_{\rm gate} =\frac{U_d-Z}{\sum_j\widetilde r_j},$$ where $j$ indexes directions and “gate” denotes this direct derivative contribution. The masked pooled feature additionally connects token scores to the direction scorer. Gather/scatter propagate gradients to selected feature values; the gates supply the surrogate for discrete selection. This biased straight-through estimator trains the selectors jointly with the RD objective. The token budget uses the batch average of soft selection probabilities: $$\mathcal L_{\rm tok} =\left(\frac{1}{B}\sum_{b=1}^{B}\frac{1}{N_b}\sum_{n=1}^{N_b}p_{b,n}-\rho\right)^2, \qquad \rho=0.5,$$ where $B$ is the number of feature maps in a minibatch, $N_b$ is the token count in map $b$, $p_{b,n}$ is the probability of retaining token $n$, and $\rho$ is the target batch mean. This soft training constraint leaves each image's retained subset and token count content-dependent. The direction loss similarly targets an average of $\kappa=2$ directions; both penalty weights are $10^{-2}$. At inference, feature-dependent thresholding and the minimum-token safeguard determine the actual count, rather than a fixed 50% quota. > **W2.3.** Entropy decoding and runtime measurement Thank you for asking about the decoding path. ASRIC decodes channel groups in order and positions in raster order within each group. One context and prototype readout serve 80 channel-specific predictions at each group–position pair. Five Gaussian components per channel produce 1,200 distribution parameters. The decoded vector then enters the context for subsequent positions; hyperprior and completed-group features are reused. Timing includes transforms, context prediction, retrieval, CDF construction, arithmetic coding and data movement. Each CLIC image uses batch size one, ten warm-ups and 50 timed repetitions synchronized at call boundaries. We average the per-image means over all 41 images. **Table R4:** Decoding-stage times (s) on RTX 4090 and A100, using CLIC images and batch size one. Strictly lower times within each platform are bold. | Stage | RTX 4090 / CMIC | RTX 4090 / ASRIC | A100 / CMIC | A100 / ASRIC | | :--- | :---: | :---: | :---: | :---: | | Hyperlatent decoding | 0.006 | 0.006 | 0.005 | 0.005 | | Hyper-synthesis | 0.002 | 0.002 | 0.002 | 0.002 | | Primary-latent entropy decoding | **0.279** | 0.295 | **0.239** | 0.252 | | Synthesis transform | 0.199 | **0.105** | 0.169 | **0.087** | | Other overhead | **0.002** | 0.003 | **0.002** | 0.003 | | Total | 0.488 | **0.411** | 0.417 | **0.349** | On RTX 4090/A100, synthesis saves 0.094/0.082 s, while entropy decoding adds 0.016/0.013 s and other overhead adds 0.001 s on each GPU. The net savings are 0.077/0.068 s. SAM reduces synthesis work by packing selected tokens into shorter sequences and scanning only active directions. Unselected features bypass scanning. The reported times include routing, packing, reordering and scattering, so the savings include the cost of sparse execution.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Responses to All Reviewers

Dear Reviewers, AC, SAC, and PC:

We sincerely thank you for your support and deeply appreciate the time and effort you have devoted to the review process. We thank all three reviewers for their careful reading and constructive feedback. We appreciate their positive feedback on routing decisions that can be reproduced at the decoder (6mLK, q1oR, tfBi), the practical value of adapting computation and entropy-model capacity to image content (6mLK), and the controlled comparisons (tfBi). We have revised the manuscript, with all changes highlighted in blue. Our main revisions and follow-up results are summarized below.

  1. Clarifying the methodological contribution. SAM retains dense local features and spatial geometry while selectively executing long-range updates. MASKR selects prototype banks for a shared causal predictor. These distinctions from generic token pruning and MoE enable content-adaptive transform computation and access to a larger predictive memory under a fixed read budget. (tfBi)

  2. Isolating the benefits of adaptive allocation. Learned Top-2 outperforms fixed, random and balanced bank selection at matched capacity and read budgets, and smaller dense dictionaries at the same read budget. The interval and boundary ablations show that using grid displacement and carrying state across scan lines improve RD. Dense SAM and joint ablation results further show better RD with adaptive allocation at similar FLOPs. (6mLK, q1oR, tfBi)

  3. Explaining sequential decoding latency. Table R17 reports primary-latent decoding times of 295.37 ms on RTX 4090 and 252.11 ms on A100, including symbol decoding and synchronization. A persistent CUDA kernel keeps the loop on the GPU to avoid repeated dispatch and transfers. Parallel attention over prototypes and probability/CDF calculations across channels shorten each step. Together, these optimizations reduce execution time while preserving the causal order of positions and symbols. (tfBi)

  4. Comparing storage and runtime. Both ASRIC configurations have active parameter counts close to CMIC's (Table R18). Our width-reduced dense SAM variant has slightly more total parameters and achieves better Kodak RD and faster coding on both GPUs. The default offers further RD gains at higher storage cost, comparable combined coding time on A100, and faster decoding on both GPUs. These configurations offer a practical choice between RD and model storage. (q1oR)

  5. Verifying meaningful and stable RD gains. Under our evaluation protocol, GLIC (CVPR 2026) and CMIC (ICLR 2026) differ by only 0.07 percentage points in Kodak BD-Rate despite their different architectures. Their close results highlight the challenge of obtaining large gains at this computational scale. ASRIC improves on them by 1.01 and 0.94 points, respectively, at comparable FLOPs, while its combined coding time is comparable to or shorter than CMIC's on both GPUs. ASRIC also outperforms CMIC in all three independent runs on every dataset. These results demonstrate meaningful and stable RD gains within a similar computation and runtime budget. (6mLK, q1oR, tfBi)

We thank all reviewers again for their valuable comments. We also revised the manuscript to improve clarity and readability. We clarified the fixed target projection, EMA prototype updates and selector gradients. We specified the training and evaluation settings and explained the statistics and color scales used in the routing visualizations. These changes make the implementation and experimental comparisons easier to reproduce. Detailed responses to each point follow.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.