- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes ASRIC, a learned image codec with two content-adaptive mechanisms. In the transforms, Spatially Adaptive Mamba blocks run the selective scan only on a learned subset of tokens and scan directions, and scale the discretization step by the L1 grid distance between consecutive retained tokens. In the entropy model, a DCAE-style learned dictionary is split into 8 banks of 16,384 prototypes. A decoder-causal query selects the Top-2 banks per spatial position and channel group, and one retrieval is shared by the group's 80 channels, so no routing indices are transmitted. All learned baselines are retrained on a 500k-image ImageNet subset. ASRIC reports BD-rates of -19.15 / -18.67 / -20.37% vs VTM-9.1 on Kodak / CLIC / Tecnick (0.3-0.9 points better than CMIC), with 4.2% fewer FLOPs and lower decoding time on an RTX 4090. Ablations cover selector policy, gap rule, bank count, prototypes per bank, Top-K and the memory update rule.
Critical strengths
-
Both routes are reproducible at the decoder without side information, and causality, CDF integerization and tie-breaking are specified carefully (A.4-A.5). This matters because adaptive entropy models easily break encoder/decoder determinism; here the design is described well enough to be implemented.
-
The comparison is controlled: all learned baselines are retrained on the same data and timed through one wrapper on the same hardware, and the ablation set is broad. This is the main reason I take the relative trends seriously despite the concerns below.
Critical weaknesses
-
The efficiency claims rest on numbers that look internally inconsistent. (a) Decoding time. Sec. 3.1 and A.4 describe a raster-order spatial context inside each group, i.e. a fully serial decode over positions. A ~2K CLIC image has about 11k latent positions at 1/16 resolution, so four groups give ~44k sequential steps. Each step includes routing, attention over 2 x 16,384 prototypes, the parameter head and arithmetic decoding of 80 symbols. A decode time of 0.41 s implies roughly 10 µs per step, about the cost of a single kernel launch. In the same table, Cheng2020-attn, which also uses a serial spatial context, takes more than 10 s. (b) FLOPs. A.4 gives about 70.87 GFLOPs per pass for Top-2 retrieval at 768x512, i.e. about 0.07 TFLOPs per extra bank for encoder plus decoder. The increments in Table 9 (+0.06 / +0.12 / +0.44 T for Top-1->2, Top-2->4, Top-2->Dense) match this almost exactly. But the protocol says FLOPs are CLIC native-resolution averages, where there are roughly 6-7x more latent positions, so dense retrieval should add about 3 T. The advantage over CMIC is only 0.10 TFLOPs (4.2%) and 0.08 s of decoding time, so if either accounting is off, the efficiency claim does not hold.
-
The RD gains are small relative to plausible noise, and the contribution is incremental. The margin over CMIC is 0.94 / 0.81 / 0.32 points, and most ablation gaps are 0.05-0.4 points, apparently from single training runs. The two mechanisms share no decisions (Sec. 3.1), and each is a modest extension: token/direction pruning for vision SSMs, and MoE-style Top-K selection over a DCAE dictionary. Tables 7 and 9 show that more memory does not help (dense retrieval is only 0.05 points better than Top-2), so the "richer memory under a bounded budget" motivation is not really supported. The sparsity savings are also measured against an expensive starting point (full-scan ASRIC costs 5.31 TFLOPs) rather than against a dense model with the same FLOPs. Without seed variance and an iso-FLOP dense baseline, I cannot attribute the gains to adaptive routing.
Questions for authors
- Which decoding schedule produced the times in Tables 1 and 6: raster order as described, a wavefront, or something else? Are the reported rates computed from bitstreams decoded with that schedule, with bit-exact reconstruction?
- At which input resolution were the FLOPs in Tables 1, 2 and 9 computed? If the Top-K increments are at 768x512, please give corrected CLIC numbers.
- Which HPCM variant (Base or Large) was retrained, and what schedule did each baseline follow? In Table 1, HPCM looks weak (-16.29% on Kodak) compared with published HPCM-Large results; how do the official checkpoints compare?
- Can you report results over 2-3 seeds for ASRIC and CMIC, and a dense SAM baseline scaled to about 2.3 TFLOPs?
6 comments
Response to Reviewer tfBi
W1.2. Input resolution and retrieval FLOPs
Thank you for identifying this discrepancy and helping us improve the manuscript. Tables 1 and 2 report FLOPs on CLIC Professional Validation. Its mean image width and height are approximately 1789 and 1189 pixels. Table R12 reports the updated CLIC counts alongside Kodak (), with units stated explicitly.
Table R12: Capacity and retrieval ablations from Table 9. Kodak and CLIC Professional Validation FLOPs are reported in G and T, respectively; Kodak BD-Rate is relative to VTM-9.1.
| Control | Variant | Total prototypes | Kodak FLOPs (G) | CLIC FLOPs (T) | Kodak BD-Rate (%) |
|---|---|---|---|---|---|
| Capacity/KB | 4,096 prototypes | 32,768 | 280.52 | 1.74 | -18.31 |
| 8,192 prototypes | 65,536 | 315.95 | 1.92 | -18.78 | |
| 16,384 prototypes (Ours) | 131,072 | 386.82 | 2.29 | -19.15 | |
| 32,768 prototypes | 262,144 | 528.55 | 3.03 | -19.12 | |
| Retrieval | Top-1 (12.5%) | 131,072 | 315.95 | 1.92 | -18.72 |
| Top-2 (25.0%) (Ours) | 131,072 | 386.82 | 2.29 | -19.15 | |
| Top-4 (50.0%) | 131,072 | 528.55 | 3.03 | -19.18 | |
| Dense (100%) | 131,072 | 812.02 | 4.51 | -19.20 |
All configurations store eight banks. Capacity/KB fixes Top-2; Retrieval fixes 16,384 prototypes per bank, with percentages indicating the fraction of active banks. Both Ours rows describe the same default model. Non-default Kodak counts are calculated from 386.82 G by adjusting prototype-attention FLOPs using the equation below and holding other operator counts fixed. Bold indicates the best BD-Rate within each ablation group.
The default Kodak count is 386.82 GFLOPs, or 0.38682 TFLOPs. Following A.4, the other Kodak entries vary the prototype dot-product and aggregation costs while holding the remaining operator counts fixed:
\frac{8Qd_m\bigl(KM-2\times16{,}384\bigr)}{10^9} \quad\text{GFLOPs},$$ where $F_{\mathrm{Kodak}}$ is the encoder-plus-decoder count, $K$ is the number of selected banks, $M$ is the number of prototypes per bank, $d_m=88$ is the prototype dimension, and $Q=4\times48\times32=6{,}144$ is the number of queries per coding pass. The factor eight counts dot products and aggregation in both encoding and decoding, with a multiplication and an addition counted separately. Thus, A.4's 70.87 GFLOPs is the Top-2 attention cost for one Kodak pass; reading one additional default bank in both passes also adds 70.87 GFLOPs. Note that the FLOPs reported in Table 9 of the original manuscript differ from those in Table R12, as the CLIC estimates used the Kodak per-bank increment. This underestimated the additional computation of reading more prototypes on CLIC. The updated counts retain the default configuration as the reference and use the increment for the corresponding dataset. CLIC counts average all 41 images after padding each image to multiples of 64; the mean dimensions above summarize the dataset rather than define one fixed input size. The conclusions of Tables 9 and R12 remain the same. At fixed Top-2 retrieval, 16,384 prototypes per bank give the best Kodak BD-Rate among the tested capacities. Increasing to 32,768 adds computation without improving RD. At fixed stored capacity, Top-2 achieves $-19.15\\%$ versus Dense's $-19.20\\%$, while reducing CLIC FLOPs from 4.51 to 2.29 T (49.2%). The main-table comparison also remains 2.29 TFLOPs for ASRIC versus 2.39 TFLOPs for CMIC. The RD scores and measured runtimes are unchanged.Response to Reviewer tfBi
W2. Stability and the contribution of adaptive allocation
W2.1. Methodological novelty and separate routing decisions
Thank you for raising this concern. CMIC adapts scan order but still updates every token; DCAE adapts attention weights but still reads the full dictionary at each query. Thus, long-range updates also run where local features suffice, and enlarging the dictionary raises retrieval cost at every location. Our motivation is to make this computation selective: SAM allocates long-range updates to content that benefits from them, while MASKR gives each context access to a larger memory under a fixed read budget. This requires preserving local reconstruction features, spatial geometry and decoder-causal probability prediction.
SAM versus token pruning. Token pruning typically scores tokens and removes a subset from later processing. SAM instead keeps dense local features and routes only the long-range branch. It adapts both token count and scan directions under average compute constraints. Packing selected tokens also changes their spacing, so SAM uses L1 displacement between retained coordinates in the state transition. Pruning-aware state alignment is relevant prior work; our geometric interval differs from scan-rank intervals at row or column boundaries. The design therefore controls which recurrent updates execute while retaining the spatial information needed for reconstruction.
MASKR versus DCAE and MoE. MASKR selects banks before prototype attention, so stored capacity can grow without reading every prototype at each query. Conventional MoE selects separate neural experts; MASKR selects prototype features for a shared probability predictor. One readout serves 80 channels with distinct likelihood parameters. Selection uses only the hyperprior and previously decoded latents, allowing both codec ends to reproduce the coding CDFs without transmitting bank indices. The contribution is selective, shared access to predictive memory within a causal entropy model.
Why the decisions are separate. A region can benefit from long-range transform processing yet be predictable from decoded context. Its token score therefore does not determine which prototype banks are useful. The selectors use transform features and decoder-available context, respectively. They are coupled through the coded latents and RD loss: SAM changes what KAEM predicts, and KAEM's code-length estimate supplies the rate term that trains SAM. Separate decisions let each stage allocate its own resource using the information available to it.
W2.2. Independent training runs
Thank you for asking about stability. We report three independent training runs for ASRIC and CMIC, each covering all five RD points. ASRIC has better BD-Rate in every run on all three datasets, with Tecnick improvements of 0.32, 0.35 and 0.44 percentage points. The training subset, validation-based checkpoint selection and each method's training settings are fixed across runs.
Table R13: BD-Rate over three independent training runs (%, against VTM-9.1). The better method is bold within each run and in the mean comparison.
| Method | Run / statistic | Kodak | CLIC | Tecnick |
| CMIC | 1 | -18.21 | -17.86 | -20.05 |
| ASRIC | 1 | -19.15 | -18.67 | -20.37 |
| CMIC | 2 | -18.27 | -17.94 | -20.12 |
| ASRIC | 2 | -19.02 | -18.55 | -20.47 |
| CMIC | 3 | -18.13 | -17.79 | -19.97 |
| ASRIC | 3 | -19.24 | -18.73 | -20.41 |
| CMIC | Mean ± SD | -18.203 ± 0.070 | -17.863 ± 0.075 | -20.047 ± 0.075 |
| ASRIC | Mean ± SD | -19.137 ± 0.111 | -18.650 ± 0.092 | -20.417 ± 0.050 |
| Mean improvement (points) | 0.933 | 0.787 | 0.370 |
Sample standard deviations use denominator , where is the number of independent runs.
ASRIC's between-run standard deviations are 0.111, 0.092 and 0.050 points on Kodak, CLIC and Tecnick; its mean improvements are 0.933, 0.787 and 0.370 points. The runs show stable training and a consistent improvement under these settings.
Response to Reviewer tfBi
Thank you for your time and detailed feedback. Below, we provide detailed responses to your questions. We would appreciate hearing whether these responses address your concerns.
Thank you for examining the decoder. The reported decoder follows channel-group order and raster order within each group. Table R10 places the main time difference from Cheng2020-attn in primary-latent entropy decoding and host overhead.
Table R10: Decoding-stage times (s) of Cheng2020-attn and ASRIC on RTX 4090 and A100, using CLIC and batch size one. Strictly lower times within each platform are bold.
Primary-latent entropy decoding includes context prediction, probability computation and entropy decoding. Other overhead covers host dispatch, transfers and synchronization waits outside those operations; non-overlapping intervals prevent double counting.
On RTX 4090, primary-latent entropy decoding takes 10.486 s for Cheng2020-attn and 0.295 s for ASRIC; other overhead takes 3.098 s and 0.003 s. On A100, these pairs are 8.953/0.252 s and 2.876/0.003 s. ASRIC's synthesis is slightly slower in this comparison, so synthesis does not explain the overall difference.
Cheng2020-attn inherits CompressAI's
_decompress_ar[1]. At every raster position, its Python loop computes context and distribution parameters, converts scale indexes to a Python list, calls CPURansDecoder.decode_stream, and copies decoded values into a tensor for the next position. With neural prediction on the GPU, this loop incurs repeated host dispatch, conversion and synchronization, whose costs accumulate across serial positions. The measured 3.098 s of other overhead on RTX 4090 is a substantial part of its total decoding time.ASRIC reuses hyperprior and completed-group features. Each group–position pair performs one retrieval shared by 80 channel predictions; the linear readout projection follows prototype aggregation. This avoids repeating retrieval for every channel and limits the work added by the prototype memory. Table R11 gives the logical operator order. ASRIC still has 4HyWy dependent steps, where Hy,Wy are latent dimensions. In the measured RTX 4090 implementation, the entropy stage and other overhead together fall from 13.584 to 0.298 s. These stages account for the large runtime difference. The shared readout reduces ASRIC's own retrieval cost; Cheng2020-attn has no prototype retrieval and already predicts its channels jointly.
Table R11: Operator order implied by the manuscript's causal decoder. Counts are logical invocations; fusion determines the number of device kernels.
Here Q=GHyWy, with G=4 channel groups and latent dimensions Hy,Wy; there are 80Q primary-latent symbols. The four stages from spatial context to distribution decoding run in order at each group–position pair, updating the vector before the next position. CDF denotes cumulative distribution function.
Both devices use ten warm-ups and 50 timed repetitions per CLIC image, synchronization at call boundaries and equal-weight image averaging. Rates count the complete bitstream, and timing covers its full decoding path. Bit-exact verification compares recovered integer latents, bank indices and integer CDFs step by step, then compares reconstruction under the same numerical backend.