- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates how to invert internal neural network features into the input domain to reveal the image structures or textual context associated with a particular layer or channel group. Unlike methods that search for an input matching a target representation alone, the proposed approach also uses source-local computational information determined by the original input. Two precalibrated linear correction modules estimate upstream states recursively along the network's computational graph.
The authors evaluate the method on several vision models and GPT-2 small. Feature selection, target swapping, and internal intervention experiments examine how the inverses relate to the source input, target features, and model predictions.
Critical strengths
-
The method is explicitly designed for reuse. Once calibrated, the same maps can support queries across inputs, layers, and channels without repeating iterative optimization for every query. This has potential value for repeated analysis of internal representations.
-
The paper provides a substantial set of diagnostic experiments. Beyond displaying inverses, the authors compare local and global maps, ablate the correction stages, swap targets and source-local operators, and intervene on internal features. These experiments help identify both the method's effects and its limitations.
Critical weaknesses
-
The research area is meaningful, but the specific problem solved and the value of the contribution remain unclear. Understanding internal representations is important, but this method requires the original input, the target features, and source-local computational information. Under this setting, the authors should more clearly explain what question the method can answer that existing feature visualization or attribution tools cannot reliably address. The paper demonstrates inversion capabilities without sufficiently establishing that the resulting outputs provide new understanding of model behavior or help users accomplish a concrete analytical task.
-
There is insufficient evidence that the inverses faithfully reflect the content of the selected features. On ResNet18, shuffling the target features while holding source-local computational information fixed barely changes similarity to the original input. This suggests that strong visual recovery may primarily derive from information supplied by the source input rather than from the target features themselves. Although this does not invalidate every channel-specific query, it weakens the basis for interpreting the inverse as showing what a particular feature extracts. Furthermore, intervening on the original feature groups establishes that those groups affect predictions, but does not directly establish that the structures displayed in their inverses accurately explain those effects. This is the central concern about interpretive faithfulness in the current version.
-
Although architecture coverage is broad, its relevance to current applications is insufficiently established. The visual experiments cover several established architectures, while the language experiments are largely limited to short-context queries in GPT-2 small. The paper lacks validation on more recent large-scale visual encoders or language models. Model age alone is not a reason to dismiss the method, but the results are clearly architecture-dependent, and some nonterminal language queries produce inverse energy at future positions. The existing evidence therefore does not establish that the method can serve as a reliable, general-purpose interpretation tool for current models; demonstrating examples in both vision and language does not replace such validation.
Questions for authors
-
What specific question does the method answer that existing interpretation tools cannot? Please use an existing experimental example to explain what additional understanding the inverse provides, beyond producing a visualization that resembles the source input more closely.
-
How should readers interpret the nearly unchanged source-input similarity after target shuffling on ResNet18? Beyond the fact that a zero target produces a zero output, what existing evidence demonstrates sufficient specificity of the inverse structure to the selected feature content?
-
What are the boundaries of the generality claim? Given the architectural differences, the short-context GPT-2 setting, and the limitations of nonterminal queries, which conclusions have been validated and which remain possible directions for extension?
2 comments
Thank you for your response. I will maintain my original rating.
Thank you for the questions about the analytical task, feature specificity and scope. We have added structural and regional-response evidence, plus DINOv2 and intermediate-node GPT-2 validation.
The analytical task and concrete findings.
Given a selected activation, the method estimates upstream states through the source input's computation. Two calibrated Wiener repairs are reused across sources, depths and channel/coordinate queries without per-query fitting or iterative search. The continuous state estimate supports structural and regional queries.
For structural reading, Final yields 34/40 correct two-query readings versus 8/40 for Raw VJP at matched sources and queries, with answers locked before source disclosure and abstentions retained. For example, a reader identifies 45-degree stripes from two selected ConvNeXt-B channel inverses, verified by the source-generation parameter. Raw VJP provides the comparison with the same source-point derivative access; G-only provides the comparison for adding the second repair.
A regional experiment connects spatial differences to channel behavior. On CUB-09352, middle-layer channels 8 and 62 place their residual energy in different windows covering foliage and the hand holding the bird. Equal-area, equal-L2 edits produce A-response shares d_A/(d_A+d_B) of 0.617 and 0.370, matching the opposite predicted preferences. Middle-layer strict crossover reaches 79.7% for peak windows versus a same-window random expectation of 20.6%, and 52.8% for foreground/background patches versus 22.2%. Both excesses pass Holm16 correction (adjusted Monte Carlo p = 0.000160). The experiment uses the existing 32-image readout-development cohort; selections were locked before new edited responses, and inference uses source-image averages.
Feature specificity and target shuffling.
The ResNet18 cosine summarizes global scene alignment under fixed source geometry. We resolve channel-dependent components by decomposing complete signed inverses, after overall normalization, into a same-layer shared direction and residual. At fixed J, changing H changes both the signed projection coefficient and residual. Mean residual changes at the two fixed geometries are 0.02984/0.03377 for ResNet18 and 0.02199/0.03733 for ConvNeXt-B in the 64-source diagnostic. Residuals retain their scale.
The new response experiment tests the predictive information in differences across channels. Middle-layer relative ordering is 83.7% for mask regions and 98.4% for peak windows. With mask regions fixed, shifting or permuting the residual maps reduces ordering to 47.0% and 53.3%; both contrasts pass Holm16. Random-channel baselines also hold the original region pairs fixed. Deep-layer ordering reaches 56.3% and 71.9% and exceeds 50% and both spatial controls after correction. Its strict-crossover excesses have adjusted p = 0.0596 and 0.1241. The supported conclusions are strong opposing regional preferences in the middle layer and relative regional correspondence in the deeper layer.
DINOv2 and intermediate-node GPT-2 validation.
We evaluated DINOv2 ViT-B/14 and complete intermediate-node GPT-2 queries against recorded upstream states:
DINOv2 uses ImageNet-40k calibration. Correct texture readouts increase from 35.94% to 65.23% on 64 controlled sources and from 17.97% to 32.23% on 64 natural sources. Calibration-budget follow-ups use the same evaluation sources and report state error and structural readability separately.
GPT-2 uses per-frequency full-channel maps calibrated on 131,072 length-16 windows, with one evaluation window per document. Queries include all positions and channels at the selected node. Adding D reduces input-embedding NMSE by 10.5% and 13.4%; mean error also improves in all nine tested target-boundary comparisons.
The validated scope covers state estimation in the tested CNN/transformer settings, visual structural readout including DINOv2, ResNet18 regional-response correspondence, and complete-node GPT-2 state recovery. For isolated nonterminal-token queries, future-position inverse energy describes statistical inverse spread; forward causal support follows the model's causal computation. Longer contexts and further partial-query semantic tasks are extensions. We will organize the revision by these questions and specify each query and calibration configuration.