acceptodds
Under review as a conference paper at ICLR 2027

Foveal Re-Encoding for Frozen Backbones

Abstract

Human vision is foveated, resolving detail only around the point of gaze and moving it wherever the task needs acuity, whereas a vision foundation model encodes one global view into a low-resolution grid and can look closer only by pushing a crop through the backbone again. Feature upsamplers do not change this, since they are trained to reproduce the backbone's encoding of the same field of view sampled more finely, which leaves the scale of analysis where the single global pass put it. We give a frozen DINOv3 a fovea. Conditioned on a point query, foveal re-encoding (FoRE) runs a lightweight modulator on the coarse map and the queried box of that same global view, predicting the features the backbone would produce by re-encoding the box at native resolution, in 2.8 milliseconds per query and almost six times more cheaply. It reads no pixel outside the view an image-guided upsampler receives, and what it learns is a remapping of that view's encoding rather than a reconstruction from pixels, so it differs from an upsampler in what it predicts rather than in what it reads. Distillation against the backbone's own re-encodings needs no labels or backbone updates. Across 25 datasets, more than fifteen readouts, and three domains, the predicted features are the strongest source apart from the teacher almost everywhere and lead every upsampling baseline by wide margins under in-context matching, with the largest gains where one wide-field encoding collapses most structure. On per-patch readouts they surpass even the re-encoding they were distilled from, an effect we trace to variance reduction and summarize in an averaging law for when it appears and reverses.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
vision foundation models, DINOv3, feature upsampling, foveation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_HMRpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FoRE, a local feature re-encoding method for frozen vision backbones. Through label-free distillation, it trains a lightweight modulator to predict the features that the backbone would produce by re-encoding a local region, given the global feature map, a point query specifying location and scale, and the corresponding crop of the global view. This reduces the cost of repeatedly running the backbone. Extensive experiments across datasets, backbones, and downstream readouts show that FoRE outperforms the evaluated feature upsampling methods under in-context matching and surpasses the teacher on some readouts. The paper further explains the student–teacher performance reversal through variance reduction and the internal averaging performed by the readouts.

Critical strengths

1、The paper addresses an important problem, and the method is well motivated. FoRE provides frozen vision backbones with an interface for querying local high-resolution features by location and scale, offering clear practical value.

2、The experimental evaluation is broad. The authors evaluate the method across multiple domains, datasets, readouts, and backbones, and compare it with several feature upsampling baselines, strengthening the credibility of the conclusions.

Critical weaknesses

1、The evaluation primarily follows a crop-grid protocol and scores only positions within the query support. It remains unclear whether FoRE would retain its advantages on standard pixel-level tasks and within complete downstream pipelines.

2、What specific assumptions are required for the “averaging law” to hold? Could the authors distinguish rigorously provable conclusions from explanations based on empirical observations?

3、The paper attributes the performance gains to the re-encoding distillation target. However, the experiments also change the architecture and training procedure, without a controlled comparison that holds the architecture fixed and varies only the supervision target. How can the sources of the performance gains be distinguished?

Questions for authors

See Weaknesses

Reviewer_KwrROpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces FoRE, a lightweight method that lets a frozen vision foundation model effectively zoom into selected regions of an image without running the expensive backbone again. FoRE shows that a frozen vision transformer can gain an efficient, query-driven form of visual attention by learning to approximate how its own representations change when it looks at a smaller region, rather than simply increasing feature-map resolution. Downstream experiments show that FoRE in many cases achieve similar or even better than the performance of DinoV3.

Critical strengths
  1. The proposed method has impressive cross-domain generalization. The authors demonstrate that FoRE trained exclusively on aerial imagery can be directly applied to natural images and indoor scenes without any fine-tuning. Despite the substantial domain shift, the model maintains strong performance. These results suggest that FoRE learns transferable feature re-encoding mechanisms rather than relying on domain-specific visual patterns, highlighting its robustness and potential for broad applicability.
  2. This paper provide comprehensive analysis on its empirical results, and found an supervising thing that some time FoRE can out-perform than its teacher. I think this important finding might be interesting to the computer vision community in general, and the theory behind that, although this paper only briefly covers, might be valuable.
Critical weaknesses
  1. FoRE relies heavily on the global backbone features to predict local re-encoded embeddings. Although the proposed spatial gating ensures that feature modifications are confined to the queried region, it does not prevent the resulting embeddings from being influenced by irrelevant background regions, since the global ViT features may already incorporate information from the entire image through self-attention. This raises concerns for anomaly detection, where preserving subtle and rare local deviations is particularly important. Moreover, the paper's proposed canonicalization effect may further suppress anomaly-specific information by reducing feature variations that are not consistently shared across samples, potentially making anomalous regions appear more similar to normal ones. While the authors demonstrate strong performance on various downstream tasks, they do not investigate whether FoRE preserves such fine-grained anomalous characteristics or how sensitive its predictions are to changes in the surrounding background. A controlled experiment that keeps the queried region fixed while varying the background, together with evaluations on anomaly detection benchmarks, would help clarify this potential limitation.
  2. Some important related work are missed, such as FOVI [1], Zooming without Zooming [2], Grasp Any Region[3], please discuss them, especially for FOVI. In my understanding FoRE and FOVI are solving the same task, and therefore an apple-to-apple comparison might be required to truly establish the position of FoRE.

[1] FOVI: A biologically-inspired foveated interface for deep vision models, ICML 2026

[2] Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perception, ICML 2026

[3] Grasp Any Region: Towards Precise, Contextual Pixel Understanding for Multimodal LLMs, ICLR 2026

Questions for authors
  1. While the paper reports strong average feature similarity and extensive downstream results, it doesn't provide a systematic analysis of the distribution of reconstruction errors across individual queries. In particular, it remains unclear whether a small subset of challenging queries, such as those involving tiny objects, fine-grained structures, or unusual visual patterns, suffers from disproportionately large errors. Reporting per-query error distributions, high-percentile errors (e.g., P90/P95/P99), and their dependence on query scale would provide a more comprehensive assessment of FoRE's reliability and potential failure modes. Could you please present such result to provide better context on the paper?
Reviewer_KMCHOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces FoRE: a method for predicting the features a frozen vision encoder E would produce when processing a selected region. E first processes a resized whole image to produce a global, but coarse, feature map. A smaller network (referred to as a "Modulator") M then takes this map + the selected region’s location and size, and a crop from the resized image to predict features for that region.

During training, E acts as a teacher and processes crops from the original image, to generate target features. M learns to match these targets using a cosine-distance loss. The method does not require image labels.

At inference, the global feature map is reused across queries, avoiding a full pass through E for each crop. M contains three convolutional blocks that process the coarse features. The query's location and size are used to condition M.

A coordinate-based decoder combines features from a 3x3 neighborhood, spatial coordinates, query information, and convolutional features extracted from the image crop. It uses a multilayer perceptron (MLP) to predict corrections to interpolated features. Binary masks restrict the learned residual updates to a disc around the query. M has approximately 9.2 million trainable parameters.

Critical strengths

FoRE reduces the cost of examining several regions in an image by reusing one global encoding. The method pays off (relative to full execution through E) when we have 2 or more queries.

The method keeps the existing backbone frozen and does not require labeled training data.

The evaluation of the method is comprehensive, and spans through multiple backbones, 25 datasets, readouts, and baselines. Experiment results show strong results on several segmentation tasks.

Critical weaknesses

The paper is a painful read, which makes the method hard to understand. The introduction reads like a disconnected sequence of sentences. Several terms need clearer explanations when first used. For example, paragraph 2.1 refers to a "teacher grid" before defining the teacher output and its spatial grid. "Native resolution" is also ambiguous, since the teacher crop is resized. Explaining the teacher and student image inputs earlier would make the method easier to follow.

An important related method is missing. FeatSharp (ICML 2025) already combines global features with features from separately encoded image tiles. FoRE should discuss this connection and compare accuracy and computation cost against it.

The training target may lose global context. The teacher processes each crop separately, so it cannot see the surrounding scene. Although FoRE receives global features, matching this teacher does not guarantee that its predictions preserve useful global information.

The explanation for the gains is not fully demonstrated. FoRE changes two things at once: what the network learns to predict (FoRE directly matches features produced by encoding a zoomed-in crop); and how it makes that prediction: FoRE uses a decoder that can learn changes to feature vectors. The experiments do not clearly separate the benefit of the training target from that of the more flexible decoder.

The proposed "averaging law" reads a bit "handwavy" and would benefit from stronger evidence.

Questions for authors

Could you better describe the accuracy/cost tradeoff? Bilinear interpolation is cheapest, but not most accurate. It would be useful to plot the Pareto frontier of these feature upsampling methods.

Could you compare the method against FeatSharp (perhaps use it as a baseline)?

Could you quantify how much global context is lost by having the zoom teacher process image crops?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.