- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper revisits binary sparse representations for interpreting vision embeddings. Instead of learning a feedforward binary encoder through a continuous relaxation or a straight-through estimator, BASICS-SAE infers a fixed-cardinality binary code through iterative atom swaps. Each accepted swap reduces reconstruction error for the current dictionary. Dictionary learning uses ridge regression with exponentially decayed sufficient statistics. The experiments compare BASICS-SAE with continuous and binary SAE baselines on CLIP and DINOv3 embeddings, with additional DINOv2 and MP-SAE results in the appendix. The evaluation covers reconstruction, prototype-based few-shot classification, distribution shift, and concept alignment. The strongest results are the improvements over the relaxation-based binary baselines and the higher coalition-level TAPAScores.
Critical strengths
The method directly optimizes the binary reconstruction problem, and the local update is well motivated. Equations (7)–(9) correctly account for the interaction between the removed and inserted atoms. This gives a concrete guarantee for each accepted swap without relying on a surrogate gradient. The ablations also support the importance of refinement: on CUB, increasing the swap budget from 16 to 128 improves NMSE from 0.0777 to 0.0568 and 10-shot accuracy from 71.62% to 74.89%.
The evaluation goes beyond reconstruction and includes relevant checks in the appendix. BASICS improves substantially over the tested binary baselines, while remaining competitive with continuous models in several settings. For example, the DINOv3 Top-10 TAPAScore increases from 0.5278 for BatchTopK to 0.6822 for BASICS. The additional DINOv2 results, MP-SAE comparison, six-seed stability experiment, and inference-latency measurements make the empirical study more useful. I also appreciate that the paper reports cases where continuous models retain an advantage.
Critical weaknesses
The few-shot protocol needs clarification. The paper reports 50-shot CUB results, although the standard split has roughly 30 training images per class. An alternative split or sampling scheme could explain this, but it is not specified. The paper should also distinguish dictionary-training, prototype, and query images, clarify whether ImageNet-A/R prototypes use labeled target-domain data, and report the distance metric and feature normalization. These choices affect how the few-shot results should be interpreted.
The optimization guarantees go beyond what the algorithm establishes. Algorithm 2 only tries replacing the weakest active atom. If that replacement fails, it stops even when replacing another active atom could improve reconstruction, so it does not guarantee a one-swap local optimum. Similarly, combining historical codes, decayed statistics, and a separate moving-average bias update does not automatically guarantee consistent improvement of one fixed objective. The paper should distinguish the valid guarantee for individual swaps with a fixed dictionary from claims about the full online training procedure.
Questions for authors
- How are the few-shot support and query sets constructed? Please explain the CUB 50-shot setting, including the number of distinct support images per class and any resampling. Please also specify the dictionary-training split, prototype metric, normalization, and whether ImageNet-A/R prototypes use source-domain or target-domain labels.
- Does the implementation stop after testing only the weakest active atom, as Algorithm 2 states? If so, how often do other improving single swaps remain at termination? A comparison with checking all active removal candidates, at least on a smaller subset, would help establish the cost–quality trade-off.
- What objective is guaranteed to decrease during dictionary and bias updates? Please clarify how historical statistics, changing codes, ridge regularization, and the bias EMA fit into the monotonicity argument. Appendix A.1 also lists a decoder learning rate of 0.01; how is this used alongside the closed-form dictionary update?
- What exact TAPAScore protocol is used? Please identify the perturbation dataset, the matching/evaluation split, and the treatment of continuous activations. Since inference starts from a random support, are random draws controlled across each original/perturbed image pair? The six-seed experiment addresses aggregate training stability, but does not directly establish the stability of an individual image’s code under repeated inference. This distinction matters for attributing latent changes to the image intervention. Additional Comments
3 comments
W2/Q3: Objective and decoder learning rate. No single objective decreases monotonically over the full online procedure, and we will remove that claim. For a fixed dictionary and bias, every accepted swap strictly lowers the exact reconstruction loss of the current batch. The dictionary update is the exact minimiser of a ridge-regularised loss over all past batches, exponentially down-weighted by (online dictionary learning with forgetting, Mairal et al., 2010). As stated in the paper, the bias is updated by an exponential moving average (rate ) of the residual left after decoding; this is a stabilising heuristic, and we make no descent claim for it. These steps do not chain into one monotone objective: the statistics contain stale codes, each new batch reshapes the surrogate, the bias shifts the centring of later batches, and the ridge term appears only in the surrogate. The "decoder learning rate 0.01" in Appendix A.1 is mislabelled: it is the bias EMA rate . The dictionary has no learning rate, as it is set by the closed-form solve. We will correct the label.
Q4: TAPAScore protocol and inference randomness. We apply the protocol of Klotz et al. (2026) unchanged for all methods. Matching: latent–attribute matching is computed on the held-out CUB test split (5,790 images; the SAEs are trained on the training split without labels) against the CUB attribute annotations, using FBMP with and coalition sizes (Top-1 to Top-10 in Table 3). Perturbation dataset: SynCUB (Klotz et al., 2026), 2,933 original/edited pairs in which exactly one attribute is replaced by another. Continuous activations: every unit with non-zero activation counts as active, and no threshold is tuned (see our response to Rev. GUFJ, Q2).
Inference randomness. The reviewer's distinction is correct: the six-seed experiment measures training stability, not the stability of an individual image's code. In the reported results, the two images of a pair were encoded with independent random initialisations. We now quantify this effect. For a single image, re-encoding with a different seed changes part of the code: the sets of active units of two encodings () have a mean Jaccard index of , against for two random sets of units, while reconstruction error stays the same. Because TAPAScore compares the codes of an original/perturbed image pair, this variation from the initialisation adds noise to the measured latent change and can artificially degrade the scores. We therefore provide two ways to control the randomness for intervention-based metrics, both affecting only the inference stage: (i) shared random initialisation, where both images of a pair start from the same random initial set of active units, so the original stochastic inference is kept but the draw is paired; and (ii) deterministic initialisation with greedy BMP followed by the same swaps, which makes every code a fixed function of the image and matches random initialisation in quality on CUB (NMSE vs. ). Controlling the randomness only strengthens the TAPAScore results (Table H). We will add this analysis and both inference options to the paper.
Table H. TAPAScore of BASICS-SAE under different inference initialisations (CLIP ViT-L/14, CUB/SynCUB, FBMP matching; same trained model).
| BASICS-SAE inference | Top-1 | Top-3 | Top-5 | Top-10 |
|---|---|---|---|---|
| Independent random init (paper) | 0.3949 | 0.6117 | 0.6159 | 0.6208 |
| Shared random init per pair | 0.4524 | 0.6758 | 0.6923 | 0.7112 |
| Deterministic (BMP) init | 0.4290 | 0.7033 | 0.7112 | 0.7301 |
Dear Rev. wQQk,
We would like to follow up on our response and the additional clarifications and experimental results provided. We welcome any further comments or remaining concerns that we might address prior to the close of the discussion phase. Thank you once again for your time and review.
Best regards, The Authors
We thank the reviewer for the careful questions, which we answer point by point. New experiments are summarised in the comment to all reviewers; this response is split into two parts.
W1/Q1: Few-shot protocol. We will add the full protocol to the main text.
Table F. Few-shot accuracy with resampled support sets (CLIP ViT-L/14, CUB, training seed 42). Mean ± std over the default support set and 5 random draws, shared across methods. 50-shot uses all training images per class, so it varies only for BASICS-SAE, through its random inference initialisation.
W2/Q2: Behavior of Algorithm 2. Algorithm 2 is not clear enough, and we will rewrite it. The procedure applies a sequence of swaps: after each accepted swap the weakest active atom is recomputed, so many different atoms can be replaced during one encoding. At each step, however, only the weakest atom and its best replacement are tested, and the algorithm stops when this swap no longer lowers the loss or the budget is exhausted. Termination therefore does not certify a one-swap local optimum; what we guarantee, for a fixed dictionary, is that every accepted swap strictly decreases the loss.
To measure how much this matters, we added an all-pairs variant that tests every (remove, add) pair at each step, and checked at termination whether any improving single swap remains (Table G). Improving swaps do remain after Algorithm 2 for most images, but the best remaining gain is about 0.02% of the input energy. Removing them lowers NMSE by about 2% relative and leaves few-shot accuracy on par, at about 4× the inference cost. We argue that the weakest-atom rule offers a favourable balance between cost and quality, and will report the all-pairs variant in the appendix.
Table G. Weakest-atom swaps vs. all-pairs search (CUB, CLIP ViT-L/14, mean of 6 training seeds). Swaps left: share of test images for which some single swap still lowers the loss at termination. Best gain: mean gain of the best remaining swap, relative to ∥xc∥2. Cost: inference time per query batch relative to Alg. 2.