acceptodds
Under review as a conference paper at ICLR 2027

Identifying, Installing, and Removing Capabilities in Vision-Language Models

Abstract

We study the computations installed by low-rank fine-tuning (LoRA) for translation, scale, and rotation tasks on a procedurally generated spatial benchmark in which one of two otherwise identical point clouds is transformed. Closed-form solutions to these simple stimuli provide both exact algorithms and cheaper statistics that are correct over part of the parameter range but fail over the rest. When asked in what direction one cloud of points must rotate to align with another, two 7B models fine-tuned on stimuli with rotation offsets between 0° and 45° do not calculate the answer based on correspondence between points, but instead compare the apparent orientation of each cloud. The fingerprint of this is evident on a full 0–180° set of evaluation stimuli, where performance alternates between high and inverted accuracy, with period 90° on statistically round clouds and 180° on elongated ones. A parameter-free reader matches these response curves by measuring the angle between the two clouds' principal axes and wrapping it into the symmetry period, agreeing with both model families. A full-range computation can be installed by training over the full 0–180° range at 1,200 examples per task when the points in the two clouds are labeled with matching numbers (cues), or at 18,000 examples per task on cue-free stimuli, where the model obtains one correspondence from the point farthest from the centroid. We also demonstrate that measured computations can be severed from model readout. A model competent at x (left/right), y (up/down), and their XOR can be tuned to emit a single constant token on x and y ("refuse" to answer), while XOR holds at perfect accuracy. The refused model's output still flips when either sign flips, so it remains sensitive to the quantities it will not verbalize. This severing costs a fraction of the supervision that installed the initial competencies. When framed as capability removal, our experiments demonstrate that, across multiple losses designed to provide readout suppression and/or algorithm destruction, such removals are successful but not durable. Reinstallation of removed verbalization or computation requires 13 to 800 times less supervision than initial installation.

Primary area:
interpretability and explainable AI
Keywords:
vision-language models, mechanistic interpretability, LoRA, shortcut learning, capability removal, unlearning, spatial reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_2EfUOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

I cannot accurately provide a summary of the technical contributions and main research questions of this paper, as it reads incomprehensible to me after almost an hour of parsing.

Critical strengths

N/A.

Critical weaknesses

Unclear main contribution and positioning

It is hard to differentiate what the main finding of this paper is and how the authors are positioning it in the abstract and intro: is it a mechanistic interpretability problem, a capability/computer vision problem, or a failure mode they are proposing solutions for? The reader should know what to expect within the first 2 pages of reading.

Abstract and introduction are difficult to interpret and lose the reader quickly

I frankly do not understand the abstract, which is the most important summary of the main claims of this work - it is incoherent, introduces a lot of undefined terminology, and it starts with deep findings rather than a framing of what the problem studied even is, or why it matters

Related work lacks proper synthesis, and contains untraceable citations and an unclear relationship to prior work

The related work section cites papers from 2015, as well as a large number of recent papers - it feels like it oscillates all over the place, without actually providing context for how each

There are also two Anonymous citations in this submission:

Anonymous. Sophon: A procedural diagnostic for spatial reasoning in vision-language models, 2026a. Accepted at NeurIPS 2026. Benchmark and generator provided as anonymized supplementary material. Anonymous. Gain and alignment at the readout layer of vision-language models, 2026b. under review.

The first is a paper that has recently been accepted to NeurIPS 2026: https://neurips.cc/virtual/2026/loc/sydney/poster/153057 - although the OpenReview page is not open, and the paper is not on arXiv, so I can’t verify the claims referenced in the related work section here, but it appears added to the citations solely based on abstract content. Feels like an LLM related work crawl that left the author list as Anonymous, but the claims should be verified against the original paper, and the author list updated to [ Zach Gazak ⋅ Ryan Swindle ⋅ Justin Fletcher ]:

Spatial failures of VLMs. Several benchmarks document failures on simple spatial questions (Rahmanzadehgervi et al., 2024; Tong et al., 2024; Fu et al., 2025; Kamath et al., 2023); the Sophon benchmark used in this work (Anonymous, 2026a) removes semantic content so that failures cannot be attributed to scene semantics. A companion paper (Anonymous, 2026b) studies what a fine-tune must change for the frozen last five decoder layers (the reader or motor regime of Gurnee et al. (2026)) to report a quantity that the layers below them (the workspace band) already encode. Here we study which computation the fine-tune installs and how that computation can be replaced or silenced.

The second Anonymous reference is referenced as “A companion paper (Anonymous, 2026b)”, which really doesn’t tell me much apart from possibly the guess that the second paper is written by the same authors as the Sophon benchmark, or a follow-up thereof. Either way, this reference reads hallucinated because it cannot be verified as it currently stands, and the authors should have omitted it. Moreover, the submission shows “under review”, so it begs the question of where exactly the authors came across this paper — in my scientific publishing practice, I am not familiar with submissions under double-blind review ever being cited in another active double-blind submission under review. This leads me to believe that reference 2026b has potentially leaked through LLMs crawling active submissions on OpenReview, and the authors simply did not verify their citation list to exclude it:

Anonymous. Gain and alignment at the readout layer of vision-language models, 2026b. under review.

Questions for authors
  1. What is the paper's primary research question, and main claims?

  2. Can you define the terminology used throughout the paper where it appears first, or refrain from introducing dense technical terms that are yet to be defined in the abstract? Can some of what this terminology represents be explained in plain language instead? In particular, please clarify the operational meaning of the "reader or motor regime", "workspace band", and the computation that fine-tuning "installs".

I would invite you to think about the minimal amount of obfuscation to what you are trying to communicate, and whether more technicality or terminology helps or hurts that intent.

  1. What does this submission claim beyond the referenced Sophon benchmark and the "companion paper"? Please distinguish the methods, experiments, and conclusions contributed here from those inherited from either reference. Can your main claims be evaluated using the evidence provided in this submission and its appendix?

  2. Can you provide reviewer-accessible links or anonymized copies of both citations 2026a and 2026b? For Sophon, please identify the relevant files in supplementary material, and clarify if these include the paper itself, or only the benchmark and generator.

3 comments

Thank you for your review. We're sorry the submitted version was hard to follow, and we have rewritten the abstract and introduction.

Research questions and claims: The question is which computation a fine-tune installs on spatial tasks with known solutions and whether it can be removed. The paper makes three claims: 1) models tuned on rotation up to 45 degrees learn an orientation-matching shortcut which fails over the wider range of rotations. 2) a correct computation can be installed by tuning over 0-180 degrees, but the supervision cost is much higher, at 18,000 examples per task. Supplying correspondence cues in the training stimuli brings that down to 1,200. 3) Six removal losses return the rotation answer to chance, but they are not durable. The paper is a controlled study, not an interpretability method or fix.

Terminology and prior work: We were sloppy with a few terms, sorry. Terms are now defined at first use. "Installs" is defined in the first paragraph of the introduction, "workspace" in Section 3, and "reader or motor regime" is removed. The paper takes stimulus generator and choice of which layers to tune from the two anonymous citations. Every experiment and claim in the paper is novel and can be evaluated from this paper alone.

Anonymous citations: These were anonymous to enforce double blind review, as neither work had been released before submission of this paper. The benchmark codebase was also private in review at NeurIPS, and so the only way we could have it is if we were the team behind that work. It is in the supplement under sophon_bench/sophon/ (the benchmark and generator, not the paper). For you to say this is likely an "LLM related work crawl", that it "reads hallucinated", and then go so far as to dig for and publish names feels inappropriate and not in the spirit of double blind review. We ask that the names be removed from the review. The full paper that you mention, whose authors you made public, is now available on arXiv. The second paper is not yet public, but we are happy to provide it to the Area Chair on request.

The rating. Your review gives a 1 at confidence 2 and does not identify an error in any experiment or result. We ask you to name the experimental claim you find unsupported, or to reconsider the rating.

Area_Chair_qqcgOpenReview ↗

Reviewer feedback request [by AC]

Dear Reviewer,

Please kindly go through the authors' rebuttal and provide timely feedback. The discussion phase ends in 24 hours (11/18 23:59 AoE). If you have further concerns and need additional clarifications, please specify them and take the limited time the authors have into account. Thank you.

Best,

AC

I appreciate the authors taking the time to respond to my questions and offer clarifications. Unfortunately, I do not believe the paper in its current form is ready for acceptance. The main contribution as stated in the authors' response and throughout the paper seems very narrow in scope, and I still believe presentation, clarity and scientific rigor in how findings are written up is important, so my concerns about the quality of the writing remain without a paper revision. I maintain my score as stated.

Reviewer_DQmGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the identification, installation, and removal of capabilities in vision–language models using synthetic point-cloud tasks. The experiments show that fine-tuned models may rely on shortcuts, while correspondence cues and activation matching facilitate capability acquisition and transfer. Models can also suppress direct answers to component tasks while retaining relevant information and performing a composite task. By comparing reacquisition and initial acquisition costs, the authors find that all five tested removal methods lack durability, with capabilities recoverable through relatively little additional training.

Critical strengths

The paper presents a novel experimental design to investigate how VLM capabilities are acquired and at what cost, using a training-free reader to provide a plausible explanation for the observed behavior.

Critical weaknesses

The paper's primary contributions are tightly coupled to a specific geometric task: the reader, symmetry sweeps, and conflicting stimuli all require bespoke manual engineering without generalizable construction principles or cross-task validation. Meanwhile, probing, teacher distillation, and activation matching are already standard techniques.

Questions for authors

Figure 2 simultaneously alters the presence of digits in both training and test images; consequently, the observed difference in sample efficiency may partially stem from test-time correspondences reducing task difficulty. Although Table 5 provides cross-evaluation, the two model groups are trained with different sample sizes and fail to cover the critical low-data regime. Thus, this does not clarify whether the 15-fold discrepancy arises from the presence of digits in the test images.

How do the authors isolate the effects of removal durability from task difficulty and retraining conditions in this ratio? Does normalizing by acquisition cost suffice for cross-task comparisons, and can the metric's robustness and generalizability be evaluated across varying task difficulties and recovery training setups?

The experiments focus on synthetic point-cloud tasks with exact solutions. For complex visual reasoning tasks that rely on multiple existing capabilities, how can the cost of learning new capabilities be distinguished from that of accessing existing ones? Does the ratio of reacquisition to initial acquisition cost remain comparable across levels of task complexity? Evaluating tasks of increasing complexity would help establish the framework’s scope.

3 comments

Thank you for your careful read of our work. We have uploaded an improved version of our paper with your specific questions in mind, and answer directly here.

Cues in the test images: We score the cue-free model on cued images of the geometrically identical scenes, and measure accuracies of 0.50 and 0.54 at 4,000 and 6,000 examples per task. The cue-trained model is at 0.99 accuracy from 1,800 examples. Cues in the image help only a model tuned to use them. We have added Table 6 in Appendix H to cover the low-data checkpoints that Table 5 did not cover, and updated Section 5 to discuss this.

Durability or task difficulty: We ran the re-attach protocol from two checkpoints that answer scale and translation but have not yet learned rotation. They reach accuracies of 0.80 and 0.89 at 18,000 examples, never crossing the 0.90 threshold. Removed checkpoints reach accuracies of 0.90 in 24 to 1,536 examples (Section 8). Ratios presented are for one attacker configuration. We did not sweep learning rate or schedule, so a better-tuned attacker might re-attach even faster. We now state that as a limit. We report the ratio for rotation only and do not claim that it is comparable across tasks.

Task complexity and construction: Given our present experiments, we are unable to test more complex tasks, or to separate the cost of learning a new capability from the cost of accessing an existing one. We have narrowed our title and stated scope to match. This initial iteration of the method needs a task with a known exact solution, and the predictor is built by hand from it. The introduction now states this as a design choice (in line with the templated shortcut tests of McCoy et al., 2019). We agree that probing and activation matching are standard techniques, and the paper cites them as such. The contribution is the measurements made with them.

Thank you for the additional experiments and clarifications. The low-data cross-evaluation in Table 6 substantially addresses my concern about Figure 2. The cue-free model still performs poorly when evaluated on cued versions of the same scenes, which makes it unlikely that test-time cues alone explain the sample-efficiency gap. The new rotation controls also strengthen the evidence for rapid reacquisition under the recovery protocol studied. I appreciate the authors’ clarification that the ratio is specific to rotation and to the chosen retraining configuration.

My main concern remains the scope of the contribution. The experiments explore a specific synthetic geometric setting in which an exact solution is available and the predictor is constructed by hand. The paper offers limited guidance on how to construct analogous tests for broader visual reasoning tasks, particularly when success depends on combining several existing capabilities. In those settings, distinguishing the cost of learning a new capability from the cost of accessing an existing one remains an open problem. The revised title and narrower claims better reflect the evidence, but these task-specific design requirements still make the proposed approach difficult to extend to a broader range of tasks and settings. On balance, I lean toward maintaining my original score.

Thank you for your reply! We understand that your remaining concern is regarding scope (not that we have misrepresented scope, but that the work is too narrow). We agree that separating the cost of learning a new capability from the cost of accessing an existing one remains open, and we don't solve that in this paper.

We want to make two points on how the approach extends:

First, the identification test does not depend on geometry. It needs a stimulus generator with parameters that provide a correct answer, candidate shortcuts that can be calculated from the stimulus, and stimuli on which the shortcut and correct answer disagree. Any procedurally generated task supplies the first of these. On natural images the same test can be run on a filtered disagreement subset, but the outcome is an overall failure rate, not an accuracy curve. Finding candidate shortcuts is done by hand at this stage of our research, and we have not done this beyond translation, scale, and rotation.

Second, the two removal tests do not require our predictor methodology or a known exact solution. Measuring the supervision required to restore a removed capability, and checking whether tuning on other tasks restores it with no labels for the removed task, could be run on any labeled task. We ran both on rotation only. We chose our narrow scope so that we would know the model we removed from had learned a full-range computation and not a shortcut.

We understand if this does not change your assessment, and appreciate your engagement with (and improvements to) the work.

Reviewer_AFmQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines how vision-language models acquire, use, and lose capabilities through fine-tuning. Using a synthetic benchmark involving transformations of point clouds (translation, scale, rotation), the authors show that models can learn simple statistical shortcuts rather than the underlying generalizable computation. They also find that providing explicit point correspondences significantly reduces the training data needed to learn rotation tasks, and that activation matching can transfer learned computations without labels. Finally, the paper investigates capability removal, showing that model representations can retain task information even when trained not to report it, and that several unlearning methods can be reversed with relatively little additional fine-tuning.

Critical strengths
  1. Controlled Experimental Setting: The authors make good use of simple and controlled synthetic tasks with known closed-form solutions. This allows them to compare model performance to the underlying computation.
  2. Generally Convincing Results: The empirical findings are interesting, particularly the periodic accuracy patterns observed in the rotation task. This, along with the additional experiments in Appendix E, provides convincing evidence that the models do not learn the underlying general computation for a task. However, I have some concerns about whether the specific shortcut statistics learned can be uniquely identified (in weaknesses).
Critical weaknesses
  1. Potentially Overstated Claims about Identifying Computations: Figure 1 shows that a model trained on a specific range of rotations results in periodic accuracy patterns when evaluated over the full range. These patterns generally match the predictions of a proposed shortcut statistic. While the results provide convincing evidence that the model is not learning the underlying general computation, I am less convinced that they uniquely identify the specific shortcut being learned. In Section 3, the authors explicitly state, “A reader that predicts the curve of a model’s accuracy as a function of stimulus angle has identified the computation.” However, could other shortcut statistics produce similar performance curves? In Appendix E, the authors provide additional evidence that the model follows principal-axis orientation, which strengthens their argument. However, I still do not think this definitively establishes that the model computes this particular statistic internally, as it could instead rely on another statistic that is highly correlated with principal-axis orientation. Overall, I find the evidence for shortcut learning generally convincing and interesting, but I think the claim of identifying the specific computation requires further justification. I would be interested in whether the authors can distinguish the proposed statistic from alternative explanations. If I have missed something crucial in the experimental setup that guarantees identifiability, please correct me.
  2. Lack of a Cohesive Paper Narrative: To me, the most interesting contribution is the controlled investigation of whether models learn the underlying task statistic. However, I found it difficult to connect the activation-matching and unlearning results to this central finding. The activation-matching results are interesting, but it is unclear to me how transferring a learned computation supports the paper's claims about shortcut learning. Similarly, the unlearning durability experiments raise important general questions about capability removal, but their connection to identifying learned computations is also not clear to me. These three directions feel like separate research questions that could each be a paper of their own if further developed. I think the paper would benefit from either developing a clearer connection between these experiments and explaining how studying all three together results in deeper insights or narrowing its scope to focus on the strongest contribution. If I have missed a crucial connection between these three directions, please let me know.
  3. Readability: Table 1 is difficult to follow. It basically contains a large blob of text describing the experimental setup and potential shortcut statistics, which I do not think improves the reader’s understanding of the author’s setting. I think some of these details would be better suited for the appendix. A figure illustrating each transformation alongside its corresponding shortcut statistic would make the experimental setting much easier to understand.
  4. Excessive Technical Detail: I personally felt that the writing often focuses too heavily on experimental details or results rather than communicating the main insights. For example, in Section 4, the authors repeatedly report specific rotation angles, sample counts, and accuracy values when a higher-level explanation would be more useful for a main text section, and these details can be offloaded to the appendix or described meaningfully in a nice figure. I would encourage the authors to utilize main text sections to emphasize the key findings and insights.

Overall, the paper presents several interesting findings. However, these findings felt disconnected rather than telling a compelling narrative that connects shortcut learning, activation matching, and unlearning durability. Additionally, I think the presentation could be improved by focusing more on the main insights rather than experimental details, and by using clearer visualizations.

Questions for authors

My questions are included in weaknesses (1) and (2). My questions concern the identifiability of the shortcut statistic learned and the connection between shortcut learning, activation matching, and unlearning durability.

3 comments

Thank you for your careful reading of our work. We have uploaded an improved draft, and discuss the changes with regard to your review here.

Identifiability: You are right. We replaced the sentence that stood out to you as an overreach. The revised text makes clear that the identification is behavioral. It is a claim about which statistic the model's answers follow, not about what the model computes internally. The evidence rules out the exact solution and the other statistics that we list. On 128 scenes built so that the principal axis and the exact rotation disagree, OneVision answers with the axis on 0.977 and Qwen2.5-VL on 0.781 (Appendix E). We cannot rule out another orientation statistic that matches the model's answers as well as the principal axis, or better, and the text now says so.

Lack of cohesive narrative: You are right, our initial submission did not properly connect the three parts. We have rewritten the abstract and introduction to do so. The narrative we intended follows one computation from start to finish. First, we ask what a fine-tune installed, and find a shortcut (Section 4). That gives us a test that tells a shortcut from a computation that is correct over the full range. Second, we use that test to install the full-range computation and measure what it costs, with labels (Section 5) and by activation matching (Section 6). The test is how we know the activation-matched student received the full-range computation and not the shortcut. Third, we take a model that we now know has learned the full-range computation and try to remove it (Sections 7 and 8). Without the first two parts we could not say what the removal was aimed at. The common lesson is that accuracy inside the tuned range does not show which computation a model has learned. High accuracy did not mean the full-range computation was there, and chance accuracy did not mean it was gone. We agree that each part could be developed further on its own.

Tbl 1 and Sec 4: Yes, agreed. Tbl 1 is now Appendix Tbl 7, and Section 4 has been rewritten to state findings without being overwhelmed with specifics of the measurements. Those measurements have been moved to Appendix E. We have added the figure you suggested (19, Appendix I), showing each transformation along with the shortcut statistic and stimuli on which the two disagree.

Area_Chair_qqcgOpenReview ↗

Reviewer feedback request [by AC]

Dear Reviewer,

Please kindly go through the authors' rebuttal and provide timely feedback. The discussion phase ends in 24 hours (11/18 23:59 AoE). If you have further concerns and need additional clarifications, please specify them and take the limited time the authors have into account. Thank you.

Best,

AC

Reviewer_AFmQOpenReview ↗

Response to Rebuttal

Thank you for your detailed rebuttal and for taking the time to address my concerns. I have reviewed the revised manuscript and appreciate the improvements.

Thank you for clarifying the identifiability of the learned computation. Once again, I think the evidence presented is sufficient to show that the models are not learning the underlying generalizable computation, but instead rely on some form of shortcut or heuristic. Since my main concern wasn't whether the learned computation is identifiable, but instead to avoid overstated claims, I consider this concern adequately addressed.

I appreciate the new figures and revisions to the writing, which have improved the readability and clarity of the paper.

However, I still have concerns regarding the overall narrative and significance of the contributions. I appreciate the authors' clarification of the intended narrative, and I now better understand the authors' vision for how the directions should connect. Nevertheless, I still feel that these directions are relatively separate and individually underdeveloped. I believe that focusing on one direction and exploring it in greater depth could have resulted in a stronger and more impactful paper. In its current form, the paper feels more like a broad exploration of several research questions within a controlled experimental setting, without any single result being particularly surprising or substantial enough to stand out as a significant contribution.

Overall, while I appreciate the authors' efforts and acknowledge that several of my concerns have been addressed, my primary concern regarding the depth and significance of the contributions remains. For this reason, I will maintain my original rating.

Reviewer_sp55OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how spatial reasoning capabilities are acquired, transferred, and removed in vision-language models. Using synthetic point clouds with known geometric transformations, the authors analyze whether models learn exact computations or simpler shortcuts. The paper further studies capability transfer through activation matching and the recovery of removed capabilities through additional fine-tuning.

Critical strengths

(1) Understanding how VLMs acquire and retain capabilities is an interesting research direction.

(2) The paper provides controlled experiments to distinguish geometric reasoning from shortcut learning, particularly through rotation symmetry analysis.

(3) The experiments include several controls and investigate capability acquisition, transfer, and removal from different perspectives.

Critical weaknesses

W1: It is unclear what broader research question is answered by studying the installation and removal of simple geometric capabilities on synthetic point clouds. The practical implications are limited.

W2: Insufficient comparison with prior work. The paper lacks systematic comparisons with existing mechanistic interpretability and representation distillation methods. Although several removal objectives are evaluated, comparisons with broader unlearning approaches remain limited.

W3: Scope and rigor do not support the title’s general claims. All results come from one synthetic benchmark, three similarity transforms, two 7B models, and a single LoRA configuration (r = 32 on decoder layers 0–22, vision tower frozen). Both installation and removal act through the same low-rank adapter on the same layers.

W4: Limited generalizability. Most experiments involve synthetic geometric transformations, and the capability transfer and removal studies are mainly conducted on one 7B model.

Questions for authors

(1) How do the proposed diagnostic experiments differ from existing shortcut-learning and mechanistic interpretability approaches?

(2) The activation-matching experiments should be compared with standard knowledge distillation methods to better establish their contribution.

(3) More baselines and prior works on VLM interpretability and representation distillation should be discussed and considered.

(4) The removal experiments need stronger controls to distinguish the recovery of previously learned capabilities from ordinary fine-tuning.

2 comments

Thank you for the time you put into your review. We've uploaded an improved version of the paper and discuss changes based on your review here.

Response to weaknesses:

W1: "Broader research question": The paper has a rewritten abstract and introduction to make this more clear. We've studied an extremely narrow ("limited practical implications") problem because we can exercise maximal control. While the end goal is to move towards more general problems, we started where we could control as much of the experiment as possible.

W2 + Q3 "baselines": We've added RepNoise and tamper-resistance training to our removal comparison (Section 8). Both re-attach rapidly after removal.

W3 and W4: Agreed on the title. We have narrowed the title scope to "Identifying, Installing, and Removing Spatial Computations with Known Solutions in Vision-Language Models", and were able to repeat the Qwen2.5-VL runs with 3 seeds, showing correlations between our predictor and the model's accuracy curve of 0.94-0.97 at every seed (Appendix E). We did not vary LoRA rank or the tuned layers, and our limitations section states that we use one adapter recipe. Sections 5-8 use Qwen2.5-VL only, which the paper now states.

Q1: "Difference from existing literature": Existing shortcut tests build stimuli on which a shortcut gives the wrong answer and report the failure rate. Because our tasks have exact solutions, we can go further. We compute each candidate shortcut from the images and use it to predict model accuracy at every rotation angle, with no fitted parameters. The model's measured accuracy follows the orientation shortcut's prediction, including the angles where it inverts. In this work, we're not suggesting a new interpretability method.

Q2 "distillation": We did not run a logit-distillation arm. We compare against a matched baseline (with the matching weight at zero), and a scrambled-target control (Section 6 and Appendix B).

Q4: "recovery against ordinary tuning": This control is now in Section 8. Checkpoints that have not learned rotation do not reach our accuracy threshold of 0.90 in 18k examples, while removed checkpoints reach it in 24 to 1,536 examples.

Area_Chair_qqcgOpenReview ↗

Reviewer feedback request [by AC]

Dear Reviewer,

Please kindly go through the authors' rebuttal and provide timely feedback. The discussion phase ends in 24 hours (11/18 23:59 AoE). If you have further concerns and need additional clarifications, please specify them and take the limited time the authors have into account. Thank you.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.