- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes AFTERIMAGE, a small adapter inserted into a frozen multimodal model that passes its contribution through discrete codes from a shared codebook, so the information it adds can be read and edited. After training, the authors find code patterns linked to features like age, diagnosis, or robot task, and show that swapping these patterns changes the output. They validate it on brain MRI generation, robot manipulation (LIBERO), and video QA (CLEVRER), reporting equal or better performance than baselines with under 1% extra parameters.
Critical strengths
The main strength is the strong performance under distribution shift with very few added parameters nearly 0.2-0.71%. On LIBERO-Pro, AFTERIMAGE beats LoRA and the continuous variant on almost every shifted suite for both robot backbones. This matters because it suggests the adapter is useful even beyond its interpretability goal. The second strength is that the paper tests its codes with interventions, not just correlations. The same-scene task swap redirects the robot to the other task, while a neutral replacement never does, which gives some idea that the codes carry information the downstream model uses.
Critical weaknesses
Major :
-
Assume I am doing lora fine-tuning of a big model and then I want to know what LoRA changed to improve accuracy. But for that why I need to trained the authors proposed interpretability again which used codebook kind of things. Because in my case I may not want codebook kind of things. Please discuss this limitation.
-
The main contribution is not clear. Is the paper proposing a better adaptation method or an interpretability method? Both are claimed, but neither is fully developed. The scope is also unclear, is it meant for image/video generation, for VLMs/LLMs, or only for "fusion" pipelines?
-
If this is an adapter paper, the baselines are not enough. LoRA is missing for MRI and no other PEFT methods are compared. With residual routing, does adding the adapter hurt the backbone’s general abilities, like - baseline model accurecy on medical, trained the model on robotics tasks and then check how much accurecy dropped on the medical tasks.
-
In Table 2, the OOD gains are very large (e.g., SmolVLA Language-Spatial 43.8 -> 93.8), even though the model never saw these shifts in training and the IID gains are small. I would expect an adapter to mainly help IID. Why does it help OOD this much, especially with the action expert frozen? Please give an explanation or analysis.
-
If the authors want to position their paper in interpretability domain, then they should plot some visualizations. In Figure 12, I do not clearly see AFTERIMAGE being better than LLM-LDM; in some columns LLM-LDM looks similar or better.
-
The paper argues that probes and SAEs are weaker, but never compares against them on the same labels and data. Why trigrams (Sec. 2.4)? The paper motivates ordered sequences as carrying composition (Sec. 2.2), but never compares trigrams with single codes or bigrams. If single codes separate the groups equally well, the ordering design (causal mask, consecutive-code motifs) is not needed. Please justify this choice with a simple comparison.
Minor :
- what authors mean by 'we extract code motifs ...' in abstract, please present the paper properly what they want to say, else it hard to understand the paper. Things are introduce before define those.
Questions for authors
see the weakness part of the paper
8 comments
Response to Reviewer Nbsd (2/3)
- In Table 2, the OOD gains are very large. Why does it help OOD this much, especially with the action expert frozen?
On why the OOD gains are large. The robustness comes from decomposing the fusion into reusable units of domain knowledge, the motifs. Each motif records one feature, such as the task, a scene object, or an action phase, and the shared codebook reuses the same motifs across inputs and depths (Sections 2.2 and 2.4). Under a shift, the surface form of the instruction or layout changes while the features defining the task remain, so the policy receives the write it was trained on. Figure 5 shows the motifs selected on training rollouts stay selective on paraphrased episodes, and Appendix K.4 shows that episodes carrying the instructed task motif succeed far more often than those without it (96.7% against 70.3% on π0.5). With the pretrained backbone plus Afterimage, the burden of multimodal understanding falls on Afterimage, and the frozen action expert keeps the lighter job of mapping a recognized state to actions. Under shift, the task, objects, and phase are still recognized in the codes (Figure 5), so the expert receives a state it already knows how to act on.
- They should plot some visualizations. In Figure 12 (It's Figure 13 now), I do not clearly see AFTERIMAGE being better than LLM-LDM; in some columns LLM-LDM looks similar or better.
We thank the reviewer for these comments.
First, on the differences between methods in Figure 12 (Figure 13 in the revised version). Any single column is a plausible scan for every method, and the comparison lies in the progression across columns, where the real row shows peri-hippocampal and ventricular fluid increasing steadily from CN 20 to AD 90. Four points separate the methods.
- The age progression. The green regions in the Afterimage maps grow along the row in step with the real row, whereas LLM-LDM shows a similar amount of green in every column and MLP-LDM changes little between CN 60 and AD 90.
- The young condition. At CN 20, Afterimage stays close to the real reference, whereas LLM-LDM already shows the fluid pattern of an older scan and MLP-LDM shows a dense red map far from the reference.
- The ventricles. Afterimage ventricles enlarge from CN 20 to AD 90 as in the real scans, whereas the LLM-LDM ventricles stay at a similar size across conditions.
- The counterfactual edit (boxed CN 80 → AD). Afterimage adds the AD-related fluid to the real CN 80 scan while keeping its anatomy, whereas the LLM-LDM edit differs little from its own CN 80 sample.
The same pattern is measured in Table 1 as age MAE and sex accuracy and in Figure 14 as regional volumes. We will revise the caption to direct the reader to the progression across columns.
Second, the MRI experiment already includes interpretability analyses under H2 and H3 (Section 3.1, Appendix H). Figure 3 and the permutation null show that the motifs selected for diagnosis, age, sex, and regional volume stay selective on withheld inputs beyond what label shuffling produces, and Figure 4 shows that overwriting or removing a motif moves the generated volume in the direction the motif encodes.
Third, the robot experiment visualizes which domain features the fusion carries, and in what order, over a single episode (Section 3.2). Figure 6 tracks keyframe by keyframe which task, object, and phase motifs have fired as the rollout advances, which makes the task and objects behind each predicted action readable and localizes where the behavior departs from the task (SmolVLA in Figure 22). Figure 5 shows these motifs stay selective on withheld and paraphrased episodes.
Response to Reviewer Nbsd (3/3)
- Why trigrams (Sec. 2.4)? The paper never compares trigrams with single codes or bigrams. If single codes separate the groups equally well, the ordering design (causal mask, consecutive-code motifs) is not needed. Please justify this choice with a simple comparison.
First, we have added a motif-length comparison (Figure 21, new). On π0.5 tasks and action phases, we repeat the motif selection with 1-grams to 4-grams and evaluate on train, held-out, and Language-shift episodes. Single codes and bigrams fire almost everywhere and do not isolate a task or a phase, trigrams concentrate on the diagonal and mark each feature selectively, and 4-grams keep the selectivity but thin the presence, especially on action phases. Trigrams are therefore the shortest run that separates the groups.
Second, the paper already tests the ordering design directly (Section 3.4). Permuting the codes at inference is about as damaging as replacing them at random on brain MRI and robots (FID 3.22 against 3.02 on brain MRI, 22.1% against 24.8% Language success on π0.5), so the write depends on which code occupies which slot. Retraining without the causal mask lowers success on both policies, most under distribution shift (80.3% against 89.2% on LIBERO-Goal under the Language shift for π0.5; Table 11, Appendix L), and its motifs are less selective on held-out and paraphrased episodes (Figure 20). Order therefore carries information the policy uses, which single-code motifs would discard.
- The paper argues that probes and SAEs are weaker, but never compares against them on the same labels and data.
We have added Figure 29, which compares a linear probe and a sparse autoencoder (SAE) with the Afterimage record on the same hidden states, task labels, and rollouts, under the LIBERO-Spatial Task shift of π0.5. Both readings use the hidden states entering VLM layer 16, the layer Afterimage reads, and the same 40k demonstration frames. The probe is a 40-way linear head, and the SAE is labelled by assigning each task its most selective feature with the protocol used for the motifs (Appendix H) and decoding each chunk by the most active of these features. On the same Task-shift rollouts, the probe predicts the instructed task on 55% of chunks and the SAE decoder on 30%, both below the motif-task alignment of Figure 18 on the same shift. Both labels are also decoupled from the outcome. The probe keeps the instructed task through a failed rollout and flips once in a successful one, and the SAE label moves among several tasks within a single episode whether or not it succeeds. The Afterimage record marks the task objects and the action phases reached so far, and the failed episode is readable at the frame where the drop replaces the hold after the carry.
Things are introduce before define those. (in abstract)
We have updated the abstract to be more clear.
Response to Reviewer Nbsd (1/3)
We thank the reviewer for the thoughtful review.
We would like to clarify the role of the codebook and how Afterimage applies to the reviewer's scenario.
The codebook is not the target. It is the material through which the domain features are read. Afterimage finetunes a pretrained model to a domain, as LoRA does, with the added ability to show which domain features take part in each prediction. The codebook turns the model's contribution into discrete units that can be named, counted, and followed across depths, which LoRA leaves implicit, and this transparency is what lets an expert trust a prediction in high-stakes domains such as medical imaging and robot control (Section 1, Appendix B).
Afterimage can replace LoRA for interpretable domain finetuning, or be trained on top of an existing LoRA to read part of the adapted computation.
We thank the reviewer for these three points and address them in turn.
First, on the scope. Afterimage targets any pipeline in which a frozen pretrained backbone fuses heterogeneous inputs into a representation that a downstream model consumes (Section 2.1). Image generation, VLA policies, and VLM question answering are three instances of this setting, and the method is not limited to them. Afterimage is an interpretability method, and performance (H1) is a constraint. The contributions lie in the causal relation between motifs and features in H2 and H3 (Sections 3.1 and 3.2, Figure 4, Table 12), while Afterimage still matches or improves performance in all three domains (Section 3).
Second, on LoRA for MRI and other PEFT methods. We have added a LoRA arm on brain MRI (Table 1, updated) and a bottleneck adapter arm on CLEVRER, and Afterimage leads both. LoRA is the appropriate reference, since it is the standard choice for adapting a frozen backbone, leaves the pretrained weights untouched as Afterimage does, and is matched in trainable parameters, so the comparison tests how the added capacity is used.
Third, on whether the adapter harms the backbone's general capability. We would like to emphasize that Afterimage is designed for domain-specific adaptation, where it improves over the frozen backbone for a specific task, so some loss on off-domain benchmarks is expected, as with any single-domain finetuning. To quantify it, we evaluated the frozen backbone and both adapted CLEVRER arms on MMLU-Pro, a ten-option multiple-choice benchmark spanning 14 disciplines (1,200 questions, identical prompt, decoding, and training step).
Afterimage stays within two points of the frozen backbone and changes fewer answers in either direction than the parameter-matched LoRA, which loses four times as much, while reaching higher in-domain accuracy on CLEVRER. With the write disabled, the model is the frozen backbone.
We hope these clarifications and additional results address the reviewer's concerns.