acceptodds
Under review as a conference paper at ICLR 2027

From Preimage Search to Source-Grounded Feature Inversion

Abstract

Feature inversion seeks to reveal what an internal representation extracts from a given input. When multiple inputs match a target representation, canonical preimage search alone does not specify the generating sample's inverse. We recast inversion as source-grounded upstream-state estimation along the target-generating directed acyclic graph, conditioned on source-local network geometry. At each boundary, a closed-form matrix Wiener map converts a mean-seed vector–Jacobian product from an adjoint signal into an upstream-state estimate. A second Wiener map estimates remaining state error from a pulled-back Jacobian–vector-product forward-consistency residual. Repaired states compose in one finite reverse pass. Fixed calibrated zero-intercept maps serve new inputs, depths, and tested channel–coordinate queries without query-specific optimisation. The formulation spans tensor components and visual distributions across convolutional networks and vision Transformers. Target–operator controls show that selected features and source-local geometry jointly determine architecture-dependent inverses. For GPT-2 small terminal-token queries, the same construction yields continuous input-embedding inverses without a text decoder. At an exploratory layer-6 contextual peak, they show semantic-cue selectivity and number-agreement routing. Across modalities, prediction-conditioned atlases display selected input-domain structure through inversion; independent interventions on corresponding original representations provide causal evidence of their decision effects. Results support source-grounded feature inversion as a reusable interface for interpreting internal representations.

Primary area:
interpretability and explainable AI
Keywords:
Feature Inversion, Model Interpretability, Matrix Wiener Filtering, Computational DAGs

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_jRuXOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates how to invert internal neural network features into the input domain to reveal the image structures or textual context associated with a particular layer or channel group. Unlike methods that search for an input matching a target representation alone, the proposed approach also uses source-local computational information determined by the original input. Two precalibrated linear correction modules estimate upstream states recursively along the network's computational graph.

The authors evaluate the method on several vision models and GPT-2 small. Feature selection, target swapping, and internal intervention experiments examine how the inverses relate to the source input, target features, and model predictions.

Critical strengths
  1. The method is explicitly designed for reuse. Once calibrated, the same maps can support queries across inputs, layers, and channels without repeating iterative optimization for every query. This has potential value for repeated analysis of internal representations.

  2. The paper provides a substantial set of diagnostic experiments. Beyond displaying inverses, the authors compare local and global maps, ablate the correction stages, swap targets and source-local operators, and intervene on internal features. These experiments help identify both the method's effects and its limitations.

Critical weaknesses
  1. The research area is meaningful, but the specific problem solved and the value of the contribution remain unclear. Understanding internal representations is important, but this method requires the original input, the target features, and source-local computational information. Under this setting, the authors should more clearly explain what question the method can answer that existing feature visualization or attribution tools cannot reliably address. The paper demonstrates inversion capabilities without sufficiently establishing that the resulting outputs provide new understanding of model behavior or help users accomplish a concrete analytical task.

  2. There is insufficient evidence that the inverses faithfully reflect the content of the selected features. On ResNet18, shuffling the target features while holding source-local computational information fixed barely changes similarity to the original input. This suggests that strong visual recovery may primarily derive from information supplied by the source input rather than from the target features themselves. Although this does not invalidate every channel-specific query, it weakens the basis for interpreting the inverse as showing what a particular feature extracts. Furthermore, intervening on the original feature groups establishes that those groups affect predictions, but does not directly establish that the structures displayed in their inverses accurately explain those effects. This is the central concern about interpretive faithfulness in the current version.

  3. Although architecture coverage is broad, its relevance to current applications is insufficiently established. The visual experiments cover several established architectures, while the language experiments are largely limited to short-context queries in GPT-2 small. The paper lacks validation on more recent large-scale visual encoders or language models. Model age alone is not a reason to dismiss the method, but the results are clearly architecture-dependent, and some nonterminal language queries produce inverse energy at future positions. The existing evidence therefore does not establish that the method can serve as a reliable, general-purpose interpretation tool for current models; demonstrating examples in both vision and language does not replace such validation.

Questions for authors
  1. What specific question does the method answer that existing interpretation tools cannot? Please use an existing experimental example to explain what additional understanding the inverse provides, beyond producing a visualization that resembles the source input more closely.

  2. How should readers interpret the nearly unchanged source-input similarity after target shuffling on ResNet18? Beyond the fact that a zero target produces a zero output, what existing evidence demonstrates sufficient specificity of the inverse structure to the selected feature content?

  3. What are the boundaries of the generality claim? Given the architectural differences, the short-context GPT-2 setting, and the limitations of nonterminal queries, which conclusions have been validated and which remain possible directions for extension?

2 comments

Thank you for the questions about the analytical task, feature specificity and scope. We have added structural and regional-response evidence, plus DINOv2 and intermediate-node GPT-2 validation.

The analytical task and concrete findings.

Given a selected activation, the method estimates upstream states through the source input's computation. Two calibrated Wiener repairs are reused across sources, depths and channel/coordinate queries without per-query fitting or iterative search. The continuous state estimate supports structural and regional queries.

For structural reading, Final yields 34/40 correct two-query readings versus 8/40 for Raw VJP at matched sources and queries, with answers locked before source disclosure and abstentions retained. For example, a reader identifies 45-degree stripes from two selected ConvNeXt-B channel inverses, verified by the source-generation parameter. Raw VJP provides the comparison with the same source-point derivative access; G-only provides the comparison for adding the second repair.

A regional experiment connects spatial differences to channel behavior. On CUB-09352, middle-layer channels 8 and 62 place their residual energy in different windows covering foliage and the hand holding the bird. Equal-area, equal-L2 edits produce A-response shares d_A/(d_A+d_B) of 0.617 and 0.370, matching the opposite predicted preferences. Middle-layer strict crossover reaches 79.7% for peak windows versus a same-window random expectation of 20.6%, and 52.8% for foreground/background patches versus 22.2%. Both excesses pass Holm16 correction (adjusted Monte Carlo p = 0.000160). The experiment uses the existing 32-image readout-development cohort; selections were locked before new edited responses, and inference uses source-image averages.

Feature specificity and target shuffling.

The ResNet18 cosine summarizes global scene alignment under fixed source geometry. We resolve channel-dependent components by decomposing complete signed inverses, after overall normalization, into a same-layer shared direction and residual. At fixed J, changing H changes both the signed projection coefficient and residual. Mean residual changes at the two fixed geometries are 0.02984/0.03377 for ResNet18 and 0.02199/0.03733 for ConvNeXt-B in the 64-source diagnostic. Residuals retain their scale.

The new response experiment tests the predictive information in differences across channels. Middle-layer relative ordering is 83.7% for mask regions and 98.4% for peak windows. With mask regions fixed, shifting or permuting the residual maps reduces ordering to 47.0% and 53.3%; both contrasts pass Holm16. Random-channel baselines also hold the original region pairs fixed. Deep-layer ordering reaches 56.3% and 71.9% and exceeds 50% and both spatial controls after correction. Its strict-crossover excesses have adjusted p = 0.0596 and 0.1241. The supported conclusions are strong opposing regional preferences in the middle layer and relative regional correspondence in the deeper layer.

DINOv2 and intermediate-node GPT-2 validation.

We evaluated DINOv2 ViT-B/14 and complete intermediate-node GPT-2 queries against recorded upstream states:

Model / targetSourcesG-only input NMSEFinal input NMSE
DINOv2 / block 6 output128 images0.432480.40165
DINOv2 / final norm output128 images0.798700.58921
GPT-2 / residual block 364 documents0.476990.42684
GPT-2 / attention module 664 documents0.533790.46237

DINOv2 uses ImageNet-40k calibration. Correct texture readouts increase from 35.94% to 65.23% on 64 controlled sources and from 17.97% to 32.23% on 64 natural sources. Calibration-budget follow-ups use the same evaluation sources and report state error and structural readability separately.

GPT-2 uses per-frequency full-channel maps calibrated on 131,072 length-16 windows, with one evaluation window per document. Queries include all positions and channels at the selected node. Adding D reduces input-embedding NMSE by 10.5% and 13.4%; mean error also improves in all nine tested target-boundary comparisons.

The validated scope covers state estimation in the tested CNN/transformer settings, visual structural readout including DINOv2, ResNet18 regional-response correspondence, and complete-node GPT-2 state recovery. For isolated nonterminal-token queries, future-position inverse energy describes statistical inverse spread; forward causal support follows the model's causal computation. Longer contexts and further partial-query semantic tasks are extensions. We will organize the revision by these questions and specify each query and calibration configuration.

Thank you for your response. I will maintain my original rating.

Reviewer_qqakOpenReview ↗
Rating
1
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper asks how to reveal, in input space, what a selected internal feature of a neural network extracts from a particular input.

It uses the original input’s local network geometry and two calibrated Wiener maps to estimate upstream states and correct their errors, composing these estimates backward through the network without optimizing each query separately.

The experiments show that the calibrated maps support repeated inversion queries across the tested vision and language models, while the recovered structures depend on both the selected features and the source input’s local geometry.

Critical strengths

1.The calibrated maps can be reused across different inputs, depths, and feature selections without optimizing each query separately. This is useful when inspecting many parts of a model, and the experiments demonstrate this reuse for both channel and coordinate queries.

2.The ablations provide evidence for the numerical benefits of the design. Local composition improves substantially over the global alternatives, and the second stage further improves state estimation and input alignment. For example, ViT-B/32 pixel cosine increases from 0.3116 to 0.7021. These comparisons help explain what the individual components contribute.

Critical weaknesses
  1. My main difficulty is understanding what would count as a successful feature inversion. The paper aims to show what a selected feature extracts from an input, but I could not find a clear way to judge whether a particular visualization captures that information correctly. The authors explain that similarity to the source image does not define correctness, yet many results emphasize this similarity. In Table 13, shuffling the terminal state leaves ResNet18's source-image alignment almost unchanged, from 0.9380 to 0.9379. This makes it difficult to tell which structures in the output can be attributed to the selected feature. The paper discusses source dependence, but its implications for interpreting an individual visualization remain unclear to me.
  2. The experiments do not clearly establish what additional understanding the inverses provide. There are many measurements of reconstruction, consistency, spatial structure, and decision effects, but I struggle to connect them to a concrete finding about feature content. In particular, the intervention experiments show that the selected feature groups affect predictions, but the groups are selected using activation times gradient and the interventions act on the original representations. Those effects do not directly test whether the corresponding inverses depict the groups' content accurately. The distinction is acknowledged in the paper, but I still need a clearer connection between the visualization and the conclusion a reader should draw from it.
Questions for authors

1.Could you walk through one representative feature query and explain what a reader should learn from its inverse, which visible structures support that interpretation, and what evidence shows that the interpretation is correct? For ResNet18, how should this interpretation account for the nearly unchanged source-image alignment after terminal-state shuffling in Table 13? 2.With the selected feature query held fixed, which result most directly shows that the full method provides a more accurate or informative interpretation than the first-stage-only method or Raw VJP? Please distinguish evidence about the inverse itself from evidence that the original feature group affects the prediction.

1 comment

Thank you for asking for explicit success criteria and a concrete example. The new experiments connect a selected feature query to a verifiable structure and a testable regional-response prediction.

Success criteria.

For a complete-node query, recorded forward states provide the reference for upstream-state error. For a specified structural query, success means that the structure is readable in the inverse and agrees with an independent label, such as stripe orientation or contour identity. For a claim about channel-specific regional sensitivity, matched local edits provide the response test. Each criterion evaluates the corresponding claim: correct structural recovery is itself a successful structural result, and regional response prediction provides an additional experimentally checked use.

Worked example: spatial prediction and measured responses.

Consider ResNet18 middle-layer channels 8 and 62 on CUB-09352. Two disjoint windows cover upper-left foliage (A) and the hand holding the bird near the bottom (B). Of the inversion-residual energy within these two windows, channel 8 places 91.3% in A, whereas channel 62 places 91.4% in B. This predicts opposite preferences under matched edits. After equal-area, equal-L2 edits, the measured feature-map changes are:

ChannelRMS change for ARMS change for BA share, d_A/(d_A+d_B)
80.0628560.0390020.617
620.0686170.1169680.370

Both preferences match the prediction. This display example was selected by the frozen source-order rule before response evaluation. Across the middle-layer peak-window cohort, 63/64 pairs have the predicted relative ordering and 51/64 show both opposing preferences. Strict crossover is 79.7%, versus 20.6% for random distinct channels on the same windows. The foreground/background scheme likewise yields 52.8% versus 22.2%. Both excesses pass Holm16 correction (adjusted Monte Carlo p = 0.000160). This new-intervention experiment uses the existing 32-image readout-development cohort; predictions were locked before new responses, and inference averages within each source image.

The concrete finding is that these selected channels have different regional sensitivities on the same input, predicted by their inversion residuals. This directly connects the displayed spatial differences to channel behavior.

Interpreting the unchanged source cosine.

Table 13 measures global scene alignment under fixed source geometry. We resolve differences within that alignment using the same-layer common direction and residual of normalized signed inverses. At fixed J, changing H changes both the shared coefficients and residuals; the two fixed ResNet18 geometries yield mean residual changes of 0.02984 and 0.03377. The regional experiment then tests the response information in differences across channels. Thus shared source appearance and predictive channel-specific differences are compatible.

Fixed-query improvements over Raw VJP and G-only.

In the two-reader study, Final yields 34/40 correct two-query structural readings versus 8/40 for Raw VJP. Answers are locked before the source is revealed, and abstentions remain in the denominator. In the separate Final/G-only comparison, both yield 34/40. The larger automatic evaluation measures the D-stage increment:

Model / structureSourcesG-onlyFinal
ResNet18 / controlled texture25687.74%93.95%
ResNet18 / natural texture12865.04%78.03%
ResNet18 / contour25698.29%98.19%
ConvNeXt-B / controlled texture25694.82%98.39%
ConvNeXt-B / natural texture12884.57%86.82%
ConvNeXt-B / contour256100.00%100.00%

These are readable-and-correct fractions over all assigned queries, averaged within source. The ResNet18 natural-texture gain is 12.99 percentage points (95% source-level CI 9.96-16.11); controlled-texture gains for both models pass the original Holm-adjusted tests. Contour performance is near saturation.

For complete-node queries, Final reduces input NMSE in all 12 historical model-target configurations. At the deepest tested targets, G-only to Final changes are 0.68263 to 0.56611 for ResNet18, 0.47051 to 0.36859 for ConvNeXt-B, and 0.71025 to 0.41751 for ViT-B/32. These measurements evaluate the inverse against state and structural references. Feature-group deletion separately measures its contribution to the classifier's decision.

We will place the success criteria and worked example before the aggregate results, making the sequence from query to visualization to experimental check explicit.

Reviewer_v13TOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a feature inversion method that uses source-dependent derivatives and two fitted linear corrections to visualize internal representations. The reusable calibration and experiments across several architectures are useful contributions. However, it remains unclear how much of the displayed structure comes from the selected feature versus the original input’s computation. Channel ablations establish prediction relevance without validating the visualizations, and the main comparisons give methods different information access. These gaps leave the central interpretability claim insufficiently supported, motivating a rejection.

Critical strengths

1.Reusable calibration. The same fitted maps support different images, layers, and feature selections without separate optimization for each query.

2.Broad experimental coverage. The method is tested across several CNN and Transformer architectures, with an exploratory extension to GPT-2.

3.Useful controls and ablations. The paper examines the two correction stages separately and includes target-shuffling and operator-swapping controls that help expose how the method behaves.

Critical weaknesses
  1. It is unclear what the selected feature contributes to the visualization. For ResNet18, shuffling the target representation barely changes similarity to the original image. This suggests that the source-dependent operators supply much of the visible structure, leaving unclear which content actually comes from the queried feature.

  2. The ablations do not validate the visualizations. Removing selected channels shows that they affect the prediction, but it does not establish that the rendered images accurately describe what those channels encode. The same ablation results would hold regardless of how the channels were visualized.

  3. The main comparison does not establish better interpretation. Iterative inversion receives the target representation, while the proposed method also uses the original input’s derivatives and calibration data. Better reconstruction alone is therefore insufficient evidence. The paper needs baselines with comparable information access and a direct test of visualization faithfulness.

Questions for authors

Could you test whether differences between channel visualizations predict differences in how those channels respond to the input? For example, suppose one channel’s visualization emphasizes a bird’s head, while another emphasizes the background. If you separately modify the head and the background, does the first channel respond more strongly to head changes, and the second more strongly to background changes? The modifications should be comparable in size and strength.

This would provide a direct check that the visualizations reveal something specific about each channel. The existing channel ablations show that channels influence the prediction, but they do not establish whether the differences shown in their visualizations accurately reflect what those channels represent.

1 comment

Thank you for proposing a direct test of channel specificity. We have completed the matched regional experiment, which links differences in the inversions to measured differences in channel responses.

Matched regional response test.

We tested two region definitions: equal-area foreground/background patches and non-overlapping 64 x 64 windows at channel-specific residual-energy peaks. Predictions use squared regional energy after removing the same-layer common direction from the signed inverses. The mask scheme selects the ten highest and ten lowest regional energy shares; the peak scheme selects up to two pairs with opposing preferences. Using the existing 32-image CUB readout-development cohort, we locked all region/channel selections before computing the new edited responses.

Edits contract pixels towards mid-gray, matching the actual changed-pixel count and perturbation L2 norm within each pair (maximum relative mismatch: 2.33e-7). For each channel, we measure feature-map changes d_A and d_B by RMS and define p = d_A/(d_A+d_B). Relative ordering requires p_A > p_B; strict crossover requires p_A > 0.5 and p_B < 0.5, directly testing both preferences in your question.

Regions / layerRelative orderingStrict crossoverSame-region random expectation
Mask / middle83.7%52.8%22.2%
Peak / middle98.4%79.7%20.6%
Mask / deep56.3%17.0%13.3%
Peak / deep71.9%21.9%16.1%

At the middle layer, 63/64 peak pairs have the predicted ordering and 51/64 satisfy both preferences. Both strict-crossover excesses pass Holm correction across 16 prespecified contrasts (adjusted Monte Carlo p = 0.000160). Relative ordering exceeds 50%, spatial-shift controls and pixel-permutation controls in all four conditions after correction. Deep-layer strict excesses have adjusted p = 0.0596 and 0.1241; its supported result is relative preference correspondence. Inference averages within each image and uses the 32 images as its units.

For the fixed-region mask test, middle-layer ordering falls from 83.7% to 47.0% under shifts and 53.3% under permutations. Peak spatial controls rerun window/channel selection; the random-channel baselines above keep each original region pair fixed. These comparisons directly test the response information carried by inversion-based channel selection.

Interpreting target shuffling.

The source cosines 0.9380 and 0.9379 summarize global alignment with the same scene under fixed source geometry. We resolve channel-dependent information through a same-layer decomposition of normalized signed inverses into a shared direction and residual. In the 64-source diagnostic, changing H at fixed J changes both the shared coefficients and residuals; mean residual changes are 0.02984/0.03377 at the two fixed ResNet18 geometries. Residuals retain their scale. The new intervention results establish that differences across channels also predict regional responses. Shared scene appearance and informative channel differences therefore coexist.

Direct validation and comparable information access.

Our fixed-query comparisons use the same source, selected activation and derivative graph. Raw VJP provides the derivative-based baseline; G-only adds the first Wiener state repair; Final adds the D error repair. These contrasts measure the value of calibrated repairs with source-point access held fixed. Target-only iterative inversion has its separately specified information setting.

In the two-reader structural evaluation, answers were locked before revealing the source. Final yields correct readings for both queried structures in 34/40 cases versus 8/40 for Raw VJP, including abstentions in the denominator. In the separate Final/G-only comparison, both score 34/40. Automatic readouts resolve additional D-stage gains: ResNet18 natural-texture success rises from 65.04% to 78.03% (paired gain 12.99 percentage points; 95% source-level CI 9.96-16.11). Complete-node tests also show lower input NMSE for Final in all 12 tested model-target configurations.

We will organize the revision around these state, structural and regional-response checks, with a worked channel pair and an explicit information-access/calibration-cost table.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.