acceptodds
Under review as a conference paper at ICLR 2027

Aperture: Training-Free Multiscale Concept Bottlenecks for Remote Sensing

Abstract

While earth observation models have advanced substantially, they still lack interpretability. While concept-bottleneck models provide interpretability and expert interaction, they are either too expensive to train for the remote sensing domain or perform poorly without annotation. We posit that in expert domains like remote sensing, such training-free models require both fine details in both image and concept space. In image space, we propose a multiscale concept bottleneck using greedy quadtree routing to locate small concepts. In concept space, we replace contrastive vision language models with pre-trained MLLMs and present a way to get reliable concept scores from them. We introduce APERTURE that combines these two components and blends concept scores at the global image and native concept-scale level to give state-of-the-art training-free model performance. To test these models, introduce SiFC, a fine-grained concept-centric dataset across three countries, with human-reviewed class-level concept maps. On SiFC, APERTURE outperforms the best training-free baselines by more than 10 percentage points in macro F1-score, and notably also outperforms supervised concept bottleneck models. Targeted component-removal tests examine whether concept scores respond to changes in visual evidence, while temporal experiments show that descriptor updates improve recognition of technological changes without retraining.

Primary area:
interpretability and explainable AI
Keywords:
Concept Bottleneck Model, Explainable artificial intelligence

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_FcK5OpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents APERTURE, a training-free concept-based recognition framework for fine-grained remote sensing imagery. The method replaces CLIP-style concept matching with MLLM-based binary concept verification, derives continuous concept evidence from Yes/No decoding probabilities, and introduces concept-specific home scales with greedy quadtree routing to recover small visual components. The paper further introduces SiFC, a concept-centric benchmark covering six industrial facility classes across three countries.

Critical strengths
  1. The problem is important and well motivated for remote sensing, where fine-grained components span substantially different physical scales.

  2. The use of an MLLM as a training-free concept verifier is simple and practically meaningful, and the use of Yes/No decoding logits is preferable to verbalized confidence.

  3. SiFC is a potentially valuable benchmark for interpretable fine-grained remote-sensing recognition.

Critical weaknesses
  1. Limited gains from the multiscale routing strategy. The empirical benefit of the proposed multiscale routing mechanism appears relatively modest. On SiFC, the global CCCD setting already achieves 78.62 macro-F1, while the full APERTURE framework reaches 80.19. Moreover, routed home-scale evaluation performs almost identically to the best fixed-crop baseline. These results suggest that the routing module contributes only marginally to the overall performance improvement.

  2. Insufficient disentanglement of individual contributions. The current experiments modify several factors simultaneously, including the use of an MLLM, structured descriptors, country conditioning, and multiscale routing. As a result, it is difficult to determine which components are primarily responsible for the observed gains. A more systematic factorial ablation would help isolate the contribution of each design choice.

  3. Lack of computational cost analysis. The proposed framework requires repeated inference with a large MLLM over multiple concepts and image regions, which may introduce substantial computational overhead. However, the paper does not report practical efficiency metrics such as inference latency, the number of MLLM calls per image, GPU usage, or computational cost. Such analysis is important for assessing the practicality of the method.

  4. Limited dataset scale and statistical robustness. SiFC contains only 800 images across six classes, with noticeable imbalance across classes and countries. Given that some of the reported improvements are on the order of only 1–2 macro-F1 points, it is unclear whether these gains are statistically robust. Reporting confidence intervals or significance tests would strengthen the empirical conclusions.

Questions for authors

Some inference prompts appear to explicitly include the class name. Could the authors clarify whether this introduces class-level priors into the concept scores, rather than making them depend solely on the corresponding visual concepts? If so, how does this affect the interpretation of APERTURE as a concept bottleneck model?

5 comments
AuthorsOpenReview ↗

Response to Reviewer FcK5: Rebuttal part 1 of 3

We thank the reviewer for the detailed comments, and we are glad the reviewer values the training-free concept verifier and the Yes/No decoding logits. We answer each point below.

W1. Limited gains from the multiscale routing strategy. The empirical benefit of the proposed multiscale routing mechanism appears relatively modest. On SiFC, the global CCCD setting already achieves 78.62 macro-F1, while the full APERTURE framework reaches 80.19. Moreover, routed home-scale evaluation performs almost identically to the best fixed-crop baseline. These results suggest that the routing module contributes only marginally to the overall performance improvement.

We agree with the reviewer that the gain in macro-F1 from routing is modest. What routing mainly adds is concept localization. Global-only scores each concept on the whole image, so it tells whether the concept is present but gives no location. Aperture keeps this global answer and also scores each concept at its home scale, so it also tells where the concept is. We checked this for one key concept per class on 185 images. Full details are provided in Appendix Table [7,8].

MethodPresentLocated
Global-only74.60.0
Aperture88.674.1

We can observe that Aperture reports the key concept in more images than global-only, 88.6% against 74.6%. It also gives a location for it in 74.1% of the images, while global-only gives none.

In comparison with the fixed-crop baseline, the accuracy is close. To see what routing adds beyond accuracy, we measured how near each view is to the concept. We marked the position (x, y) of the key concept in the images, and measured the Euclidean distance from this position to the centre of the full image, of the best-scoring fixed 2048 × 2048 crop and of the routed home-scale crop, on the images where all three report the concept.

Global distance (px)Fixed 2048 × 2048 distance (px)Home distance (px)
Mean across classes676.73858.63256.58

We can observe that the center of the full image is closer to the concept than the fixed 2048 × 2048 crop, because the images are centered on the facilities. Nonetheless, the routed home-scale crop of Aperture is the closest of the three. So although the accuracy is close to a fixed crop, routing brings the view nearer to the concept.

In summary, while the gain in macro-F1 is modest, multiscale routing also shows where each concept is, and its view is closer to the concept than the full image or a fixed crop.

W3. Lack of computational cost analysis. The proposed framework requires repeated inference with a large MLLM over multiple concepts and image regions, which may introduce substantial computational overhead. However, the paper does not report practical efficiency metrics such as inference latency, the number of MLLM calls per image, GPU usage, or computational cost. Such analysis is important for assessing the practicality of the method.

We thank the reviewer for raising this, and we agree that the paper should report these costs. We have now measured them for global-only, for Aperture and for fixed grids of crops. The table below gives the number of MLLM requests per image, the time per image and the GPU-hours, where one request is one MLLM call with one image and one question.

MethodGlobalRoutingCrop / homeTotal requestsTime (s/image)Images/hourEst. GPU-hours (800 images)
Global-only113001136.935519.1061.541
Aperture1132499645833.074108.8477.350
Global + 4 crops (2048 × 2048)113045256534.675103.8217.706
Global + 16 crops (1024 × 1024)11301,8081,92198.05836.71321.791
Global + 64 crops (512 × 512)11307,2327,345394.0279.13687.562
Global + 256 crops (256 × 256)113028,92829,0411,554.9752.315345.550

Setup. Gemma 4 31B on one NVIDIA A100 80 GB GPU per worker, BF16 precision, 224 × 224 image inputs, and a vLLM GPU-memory utilization target of 90%. The model fits on one GPU, without tensor parallelism.

As the table shows, Aperture needs about 4× the requests and 4.8× the time of global-only, so it does cost more. The other way to read small concepts is a fixed grid of crops, and this costs more at every size, from about 5× the requests of global-only for its cheapest form to about 257× for a grid at the 256 px scale. This growth matches the time complexity stated in the paper (lines 318 to 319), where an exhaustive search needs O(4ℓ)O(4^{\ell}) MLLM calls for ℓ\ell zoom levels and greedy routing needs O(ℓ+1)O(\ell + 1).

Overall, Aperture costs more than global-only, and routing keeps this cost below any fixed grid of crops.

AuthorsOpenReview ↗

Response to Reviewer FcK5: Rebuttal part 2 of 3

W2. Insufficient disentanglement of individual contributions. The current experiments modify several factors simultaneously, including the use of an MLLM, structured descriptors, country conditioning, and multiscale routing. As a result, it is difficult to determine which components are primarily responsible for the observed gains. A more systematic factorial ablation would help isolate the contribution of each design choice.

We thank the reviewer for this suggestion. Following it, we ran a factorial ablation on SiFC. The table below gives the result, where each row is one combination of the design choices.

We thank the reviewer for this suggestion. Following it, we ran a factorial ablation on SiFC. The table below gives the result, where each row is one combination of the design choices.

#SettingMLLMConcept descriptorsClass conditioningCountry conditioningMultiscale routingMacro-F1
1CLIP ViT-L/14, class name only✗✗✗✗✗62.16
2CLIP ViT-L/14, general descriptors (GCD), global-only✗✓✗✗✗41.10
3CLIP ViT-L/14, general descriptors (GCD), with routing✗✓✗✗✓46.52
4CLIP ViT-L/14, class descriptors, global-only✗✓✓✗✗42.11
5CLIP ViT-L/14, class descriptors, with routing✗✓✓✗✓44.29
6CLIP ViT-L/14, class and country descriptors (CCCD), global-only✗✓✓✓✗44.99
7Gemma 4 31B, class name only✓✗✗✗✗70.40
8Gemma 4 31B, general descriptors (GCD), global-only✓✓✗✗✗70.78
9Gemma 4 31B, class descriptors, global-only✓✓✓✗✗72.37
10Gemma 4 31B, class and country descriptors (CCCD), global-only✓✓✓✓✗75.67
11Gemma 4 31B, class descriptors, with routing✓✓✓✗✓75.62
12Aperture✓✓✓✓✓78.14

Gain (pp) from using the MLLM in place of CLIP, at the same setting. The numbers in brackets are the two rows compared.

SettingGain (pp)
Class name only+8.24 (1 → 7)
General descriptors (GCD), global-only+29.68 (2 → 8)
Class descriptors, global-only+30.26 (4 → 9)
Class descriptors, with routing+31.33 (5 → 11)
Class and country descriptors (CCCD), global-only+30.68 (6 → 10)

Gain (pp) from each other design choice, with each backbone.

Design choiceWith CLIP ViT-L/14With Gemma 4 31B
Concept descriptors−21.06 (1 → 2)+0.38 (7 → 8)
Class conditioning+1.01 (2 → 4), −2.23 (3 → 5)+1.59 (8 → 9)
Country conditioning+2.88 (4 → 6)+3.30 (9 → 10), +2.52 (11 → 12)
Multiscale routing+5.42 (2 → 3), +2.18 (4 → 5)+3.25 (9 → 11), +2.47 (10 → 12)

We can observe that the MLLM is the largest factor, with about +30 pp in every setting that uses descriptors and +8.24 pp with class names only. Country conditioning and multiscale routing add smaller gains that are positive in every comparison, +2.52 to +3.30 pp and +2.18 to +5.42 pp. Concept descriptors and class conditioning give small or mixed gains on their own.

In summary, the MLLM brings most of the gain, and country conditioning and multiscale routing each add a smaller but consistent gain on top.

Q1. Some inference prompts appear to explicitly include the class name. Could the authors clarify whether this introduces class-level priors into the concept scores, rather than making them depend solely on the corresponding visual concepts? If so, how does this affect the interpretation of APERTURE as a concept bottleneck model?

We thank the reviewer for pointing this out, and we apologize for the mistake on our side.

The class names should not appear in these prompts, since they can bring class-level priors into the concept scores. We removed them from the verification and routing prompts and ran the experiment again. As the table below shows, Aperture still improves over global-only.

DescriptorGain (pp)Paired 95% CI (pp)
HSR-CCCD, with class names (paper)+1.57[+0.41, +2.88]
HSR-CCCD, no class names (new)+2.47[+0.48, +4.48]

In short, the gain remains without class names, which suggests that the concept scores rely mainly on the visual concepts, so Aperture can still be read as a concept bottleneck model.

AuthorsOpenReview ↗

Response to Reviewer FcK5: Rebuttal part 3 of 3

W4. Limited dataset scale and statistical robustness. SiFC contains only 800 images across six classes, with noticeable imbalance across classes and countries. Given that some of the reported improvements are on the order of only 1–2 macro-F1 points, it is unclear whether these gains are statistically robust. Reporting confidence intervals or significance tests would strengthen the empirical conclusions.

SiFC is a small dataset and is not fully balanced across classes and countries. We want to clarify that SiFC focuses on fine-grained classes that are recognized through their components (lines 157 to 159), and such facilities are few. The imbalance is also present in the ground truth, where some facility types are much rarer than others. When we tried to extend SiFC, we could add only 256 images from the three countries. For example, we could add only 4 nuclear power plants and 8 LNG terminals. This suggests that a large and balanced fine-grained dataset is hard to build.

ClassOriginalAddedNow
Iron and steel plant18049229
LNG terminal49857
Nuclear power plant74478
Oil refinery1599168
Power plant180138318
Sewage treatment plant15848206
Total8002561,056

The table above shows what happened when we tried to extend SiFC. We could add only 256 images from the three countries, for example only 4 nuclear power plants and 8 LNG terminals, which suggests that a large and balanced fine-grained dataset is hard to build. Besides SiFC, the paper also has a facility subset of fMoW, with 400 images of five other facility classes from the USA, France and Russia (Table 8, line 1136), where Aperture reaches 76.55% macro-F1 (line 366).

Because SiFC is small, we also checked whether the gains are statistically robust, as the reviewer suggested. We computed paired 95% confidence intervals with a bootstrap over the 800 images (10,000 resamples, stratified by country and class, so every resample keeps the same imbalance). The interval lies above zero in every comparison of the table below.

ComparisonDiff. (pp)Paired 95% CI (pp)
Aperture − global-only, descriptors as in the paper+1.57[+0.41, +2.88]
Aperture − global-only (No class Name)+2.47[+0.48, +4.48]
Aperture − supervised CBM head, leave-one-country-out+12.35[+8.85, +15.86]
Aperture − CLIP ViT-L/14, class name only+15.98[+12.13, +19.97]
Aperture − CLIP ViT-L/14, CbD+11.72[+7.96, +15.57]
Aperture − RemoteCLIP ViT-L/14, CbD+38.81[+35.00, +42.76]
Aperture − GeoRSCLIP ViT-L/14, CbD+19.93[+16.32, +23.69]
Aperture − EarthDial 4B RGB, class name only+23.94[+20.13, +27.89]
Aperture − EarthDial 4B RGB, CbD+34.68[+30.58, +38.79]

To sum up, every interval lies above zero, which suggests that the gains, including the small gain from multiscale routing, are reliable at this dataset size.

We thank the reviewer again for the comments. Based on them, we ran the factorial ablation, measured the computational cost, and computed confidence intervals for the gains. We are happy to answer any further questions.

Reviewer_Uoo2OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes APERTURE, a training-free concept bottleneck framework for fine-grained satellite image classification. It combines MLLM-based concept scoring, class- and country-conditioned descriptors, and greedy quadtree routing for multiscale recognition. The authors also introduce the SiFC benchmark and evaluate the method on facility classification, visual interventions, and temporal adaptation.

Critical strengths
  1. The combination of global context and concept-specific local views is intuitive and does not require task-specific model training.
  2. The proposed SiFC dataset provides a useful benchmark for fine-grained facility recognition across different geographic regions.
  3. The experiments cover multiple MLLM backbones, several baselines, and additional evaluations involving component removal and descriptor updates.
Critical weaknesses
  1. APERTURE improves macro-F1 from 78.62% to 80.19% over global-only recognition, while greedy routing barely outperforms fixed-scale cropping. The key contribution of multiscale routing is limited.
  2. The SiFC dataset contains only 800 images from six industrial facility classes. Generalization to other facility categories and unseen geographic regions remains unclear.
  3. The paper claims that general-purpose MLLMs are better at concept recognition than remote sensing-specific models, but provides no direct experimental evidence. Comparisons of both CLIP-based and MLLM-based specific models (such as [1][2][3]) are lacking.
  4. The global verification and routing prompts explicitly include class names, despite the paper's motivation to avoid class-name bias in concept recognition. This raises concerns about whether concept scores reflect visual evidence or class-specific contextual cues.

[1] RemoteCLIP: A Vision Language Foundation Model for Remote Sensing. IEEE Transactions on Geoscience and Remote Sensing, 2024. [2] Rs5m and georsclip: A large scale vision-language dataset and a large vision-language model for remote sensing. IEEE Transactions on Geoscience and Remote Sensing, 2024. [3] Earthdial: Turning multi-sensory earth observations to interactive dialogues. CVPR 2025.

Questions for authors
  1. The routing prompt forces the model to select a quadrant even when the target concept is absent. How does APERTURE prevent such forced routing from introducing false-positive concept detections?
  2. All global images are resized to 224 × 224 pixels, even when using MLLMs that support higher resolutions. Would increasing the input resolution reduce the advantage of the proposed multiscale routing?
4 comments
AuthorsOpenReview ↗

Response to Reviewer Uoo2: Rebuttal part 1 of 2

We thank the reviewer for the time and the comments. We are glad the reviewer found our method intuitive and SiFC a useful benchmark. We answer each point below.

W1. APERTURE improves macro-F1 from 78.62% to 80.19% over global-only recognition, while greedy routing barely outperforms fixed-scale cropping. The key contribution of multiscale routing is limited.

While the gain in accuracy from routing is modest, we would like to clarify that its key contribution is concept localization. Global-only tells whether a concept is present. Aperture also tells where it is. We checked this for one key concept per class on 185 images. The full details are in our answer to Reviewer 1, Q1.

To show localization performance, we also quantitatively measure it for one key concept per class on a 185-image SiFC subset. We annotate both if the concept is present and, if yes, its x, y coordinates in the image. We report presence prediction accuracy and localization error. The localization error is the average distance from the center of the true concept to the center of a window where each method detects the concept.

MethodPresent (%)Distance to concept (px)
Global-only74.6676.7
Aperture88.6256.6

Aperture reports the key concept in more images than global-only model, 88.6% against 74.6%, and can also localize them better (256.6 px vs 676.7 px error), providing better ante-hoc interpretability.

Along with the localization, the gain in accuracy, though modest, is consistent across three settings. It is +1.57 pp with the descriptors of the paper, +2.47 pp with the descriptors without class names, and +2.47 pp on the extended SiFC of 1,056 images.

In summary, while the gain in accuracy is modest, we hope this shows that multiscale routing also adds concept localization, improving ante-hoc interpretability, and that the global-only variant it builds on is part of our contribution as well.

W2. The SiFC dataset contains only 800 images from six industrial facility classes. Generalization to other facility categories and unseen geographic regions remains unclear.

The SiFC is small and that generalization is an important question. As Aperture is training-free, it uses no labeled target images, no concept annotations and no parameter optimization on the target dataset (lines 212 to 214). So by default, every data point and every region in our evaluation is a generalization case for Aperture.

For the same reason, Aperture can be moved to other facility categories and regions without training, and the paper reports one such case beyond SiFC. Table 1 (line 347) includes the fMoW facility subset, with 400 images of five other facility classes (shipyard, solar farm, storage tank, water treatment facility and wind farm) from the USA, France and Russia (Table 8, line 1136), where Aperture reaches 76.55% macro-F1 (line 366).

As for the size of SiFC itself, our focus is on fine-grained classes that are recognized through their components (lines 157 to 159), and such facilities are few. When we tried to extend SiFC, we could add only 256 images from the three countries, because the ground truth lists very few of them. For example, we could add only 4 nuclear power plants and 8 LNG terminals. This suggests that a large fine-grained facility dataset is hard to build.

ClassOriginalAddedNow
Iron and steel plant18049229
LNG terminal49857
Nuclear power plant74478
Oil refinery1599168
Power plant180138318
Sewage treatment plant15848206
Total8002561,056

On this extended dataset of 1,056 images, Aperture also gets a consistent gain over global-only, 75.85% against 73.37% macro-F1 (+2.47 pp).

Overall, Aperture needs no training, so it can be used on new facility classes and regions, as the fMoW subset suggests, and it keeps its gain when SiFC is extended.

W3. The paper claims that general-purpose MLLMs are better at concept recognition than remote sensing-specific models, but provides no direct experimental evidence. Comparisons of both CLIP-based and MLLM-based specific models (such as RemoteCLIP [1], GeoRSCLIP [2], EarthDial [3]) are lacking.

We thank the reviewer for this suggestion. Following it, we ran the comparison with the three suggested models on SiFC, using the prompts and scoring of the CND and CbD rows of Table 1.

ModelCNDCbD
CLIP ViT-L/1462.1666.42
RemoteCLIP ViT-L/1429.7239.33
GeoRSCLIP ViT-L/1451.6758.21
EarthDial 4B RGB54.2043.46

On SiFC, all three remote sensing models are below the general-purpose CLIP, with both CND and CbD. Aperture, which uses the general-purpose Gemma 4 31B, reaches 78.14% macro-F1, above all of them.

AuthorsOpenReview ↗

Response to Reviewer Uoo2: Rebuttal part 2 of 2

W4. The global verification and routing prompts explicitly include class names, despite the paper's motivation to avoid class-name bias in concept recognition. This raises concerns about whether concept scores reflect visual evidence or class-specific contextual cues.

We thank the reviewer for pointing this out, and we apologize for the mistake on our side.

We agree that the class names should not appear in these prompts. We removed them from the verification and routing prompts and ran the experiment again. As the table below shows, Aperture still improves over global-only.

DescriptorGain (pp)Paired 95% CI (pp)
HSR-CCCD, with class names (paper)+1.57[+0.41, +2.88]
HSR-CCCD, no class names (new)+2.47[+0.48, +4.48]

We added a new prompt template in the paper as well. Overall, the gain remains without class names, which suggests that the concept scores rely mainly on visual evidence.

Q1. The routing prompt forces the model to select a quadrant even when the target concept is absent. How does APERTURE prevent such forced routing from introducing false-positive concept detections?

This is a fair concern, as the router always has to pick a quadrant. In Aperture, routing only decides where to look, and its output is not used as a concept score (lines 316 to 317). At the home scale, a separate verification pass asks whether the concept is visible in the selected crop and uses p(yes) as the concept score (lines 317 to 318). So the forced choice only selects a crop. If the concept is not in that crop, the concept score can answer no, and then nothing is detected.

To see if this holds in practice, we checked it for the chimney stack on all power plants and all sewage treatment plants among the 800 SiFC images. It is expected in power plants and not in sewage treatment plants, and for both the router has to pick a quadrant. The table counts the images in which the concept score pass then detects it (p(yes) of at least 0.5).

Chimney detectedChimney not detectedTotal
Power plants9783180
Sewage treatment plants6152158
Total103235338

The sewage treatment plants are the key case here, since a chimney stack is not expected in them. The final concepyt score detects it in only 6 of the 158 images (3.8%) and answers no in the other 152 (96.2%), although the router was forced to pick a quadrant in every one of them. In power plants, where it is expected, it is detected in 97 of 180 images (53.9%).

To sum up, forced routing only picks where to look, and the separate verification pass can still answer no.

Q2. All global images are resized to 224 × 224 pixels, even when using MLLMs that support higher resolutions. Would increasing the input resolution reduce the advantage of the proposed multiscale routing?

We agree that this is worth checking. So we repeated the experiment at 1,104 × 1,104 input with Gemma 4 31B and the HSR-CCCD descriptors without class names.

Input resolutionGlobal-onlyApertureGain (pp)
224 × 22475.6778.14+2.47
1,104 × 1,10482.4684.11+1.65

The table shows that a higher resolution improves both global-only and Aperture. The gain of Aperture becomes a little smaller, from +2.47 to +1.65 pp, as the reviewer expected, but it remains.

We would also like to explain why the paper uses 224 × 224. We needed one fixed input size for all methods, so that CLIP, the other baselines and all MLLM backbones are compared at the same size and the advantage of the MLLM is measured in a consistent way.

In brief, a higher resolution helps both settings, and the advantage of Aperture becomes a little smaller but remains.

We thank the reviewer again for the comments. Based on them, we ran the comparison with the three remote sensing models, repeated the experiment without class names in the prompts, and tested a higher input resolution. We are happy to answer any further questions.

Thank you for your detailed responses and additional experiments. I acknowledge the authors' efforts and have decided to raise my rating from 2 to 3. Good luck.

Thank you for going through our responses and for raising your rating. Your questions led to several of the new experiments, and we will add them to the paper. We appreciate your time and your good wishes.

Reviewer_CzL3OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes APERTURE, a training-free concept bottleneck model for remote sensing. It combines multiscale image analysis with concept scores from a pretrained MLLM to capture global context and local details. The authors also introduce SiFC, a dataset covering three countries. On this dataset, APERTURE outperforms existing training-free baselines and supervised CBMs. Further experiments examine whether concept scores reflect visual evidence and show that updating concept descriptions can help recognize technological changes without retraining.

Critical strengths

1.The paper addresses a practical challenge in remote sensing.

2.The experiments go beyond comparing with CLIP. They also compare class-name baselines, global concepts, local concepts, their fusion, and different description styles using the same MLLM. Reporting both cross-country and within-country results helps distinguish generalization issues from limitations in model capacity.

Critical weaknesses

1.The motivation highlights the lack of interpretability in remote sensing, but it is unclear how the proposed training-free CBM addresses this issue. What specific aspects of interpretability does the method improve? If these benefits come from the CBM formulation itself, the paper needs to clarify its own contribution.

2.The “home scale” strategy seems similar to approaches already used in MLLM research, such as zooming in on image regions. As presented, it appears to be an application of these existing approaches. The distinction needs to be made clearer.

3.The title suggests a broad scope in satellite remote sensing, while the experiments focus on facilities and cover only 800 industrial sites. This seems too narrow to support the scope implied by the title. A broader evaluation covering scenes such as marine environments, urban areas, and different types of facilities would help.

4.The full method improves over the global-concept variant by only 1.57 percentage points. It is unclear whether this gain justifies the additional MLLM calls.

Questions for authors

1.What specific improvements in interpretability does the proposed method provide? How do these go beyond the interpretability already offered by CBMs?

2.How does the “home scale” strategy differ from existing zoom-in approaches in MLLM research?

3.Does the 1.57-percentage-point gain over the global-concept variant justify the additional MLLM calls?

7 comments
AuthorsOpenReview ↗

Response to Reviewer CzL3: Rebuttal part 1 of 2

We sincerely thank the reviewer for the careful reading of our paper, the kind words on the problem and the experiments, and the comments that helped us make our contribution clearer.

W1&Q1. What specific aspects of interpretability does the method improve? The paper needs to clarify its own contribution.

We want to clarify that the contribution beyond the CBM is concept localization. A flat CBM tells us whether a concept is present. Aperture tells us not only whether it is present but where it is as well. The former comes from the CBM formulation; the latter is the contribution of this work.

The difference comes from how each concept is scored. A flat CBM scores each concept on the whole image, so it gives no location. Aperture keeps this global answer and also scores each concept at its home scale, so a concept found therefore comes with its location.

To show localization performance, we also quantitatively measure it for one key concept per class on a 185-image SiFC subset. We annotate both if the concept is present and, if yes, its x, y coordinates in the image. We report presence prediction accuracy and localization error. Localization error is the average distance from the center of the true concept to the center of a window where each method detects the concept.

MethodPresent (%)Distance to concept (px)
Global-only74.6676.7
Aperture88.6256.6

Aperture reports the key concept in more images than global-only model, 88.6% against 74.6%, and can also localize them better (256.6 px vs 676.7 px), providing better ante-hoc interpretability. Full details are in Appendix Table [7,8], and we will extend this evaluation to all concepts and all images in SiFC in the camera-ready version.

In summary, a flat CBM can only give a presence of a concept, while Aperture can also point to where the concept is, providing stronger ante-hoc interpretability.

W2&Q2. The "home scale" strategy seems similar to approaches already used in MLLM research, such as zooming in on image regions. As presented, it appears to be an application of these existing approaches. The distinction needs to be made clearer.

We agree that our approach and zoom-in both look at a smaller part of the image; however, there are several differences.

The first major difference is that our crop is used to build interpretable-by-design concept bottlenecks. In our method, each crop gives a concept score, and the class score is a combination of the concept scores at the global and home scale, so the decision can be traced back to each concept and each scale. In V*[1], the model takes the image tokens, the question, and the crop tokens together and makes its decision implicitly, which is significantly less interpretable.

Recent works have tested this implicit use of the crop. On the Faithfulness of Visual Thinking [2] and The Illusion of Visual Tool-Use [3] show that for crop-and-zoom models such as DeepEyes and Pixel Reasoner, even if the crops are replaced with incorrect ones, the accuracy on V* Bench changes very little. This suggests that the answer of these models does not really depend on the crop. In contrast, in Aperture the performance increases with zooming in (see our answer to Q3). Moreover, as our intervention results show the class scores drop more for the cropping model when we erase concepts from the image (Figure R1).

A second difference is how the zoom level is chosen. In zoom-in methods, the zoom level is arbitrary and covers only a few concepts. In Aperture, we know the home scale of each concept and zoom in until that level.

Overall, zoom-in methods search for a scale and tend to use the crop implicitly, while Aperture zooms to the known scale of each concept and shows each concept score, which can help the user see which concepts led to the decision.

[1] Wu, Penghao, and Saining Xie. "V*: Guided visual search as a core mechanism in multimodal llms." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.

[2] Liu, Zujing, et al. "On the faithfulness of visual thinking: Measurement and enhancement." arXiv preprint arXiv:2510.23482 (2025).

[3] Wang, Zhiheng, et al. "The illusion of visual tool-use: A causal audit of thinking with images." Findings of the Association for Computational Linguistics: EMNLP 2026. 2026.

AuthorsOpenReview ↗

Response to Reviewer CzL3: Rebuttal part 2 of 2

W3. The title suggests a broad scope in satellite remote sensing, while the experiments focus on facilities and cover only 800 industrial sites. This seems too narrow to support the scope implied by the title. A broader evaluation covering scenes such as marine environments, urban areas, and different types of facilities would help.

We thank the reviewer for this point, and we agree that the title was broader than our experiments. We have updated it to "APERTURE: TRAINING-FREE MULTISCALE CONCEPT BOTTLENECKS FOR FINE-GRAINED REMOTE SENSING RECOGNITION".

With the revised title, we aim to more clearly convey our focus on fine-grained classes distinguished by their constituent components (Lines 157 to 159). We value the suggestion of marine and urban scenes and agree that they would broaden the evaluation. However, these are often coarse-grained classes that may need less reasoning about parts (lines 152 to 153), so in this work we chose to start with facilities, where we believe concept-level interpretability can help the most.

Within facilities, we tried to make the evaluation as broad as we could. The paper also has an additional facility subset of fMoW, with 400 images of five other facility types from the USA, France and Russia (Table 8, line 1136), which gives evidence beyond SiFC.

We also tried to extend SiFC, but could add only 256 images from the three countries, because the ground truth lists very few such facilities. For example, we could add only 4 nuclear power plants and 8 LNG terminals. This suggests that a large fine-grained facility dataset is challenging to build.

ClassOriginalAddedNow
Iron and steel plant18049229
LNG terminal49857
Nuclear power plant74478
Oil refinery1599168
Power plant180138318
Sewage treatment plant15848206
Total8002561,056

On this extended dataset of 1,056 images, Aperture also gets a consistent gain over global-only, 75.85% against 73.37% macro-F1 (+2.47 pp).

Taken together, we hope the new title states our scope more clearly, and within this scope we evaluate on SiFC, on its extension and on a facility subset of fMoW.

W4&Q3. Does the 1.57-percentage-point gain over the global-concept variant justify the additional MLLM calls?

We agree that the gain in accuracy is modest and that the cost of the additional calls is a fair concern. As discussed in our answer to Q1, these calls help not only the performance but also the localization. Each concept is scored at its home scale, so the decision can be traced back to the concepts.

Besides the localization, the gain in performance, though modest, stays consistent across settings. Aperture gains +1.57 pp over global-only with the descriptors of the paper (lines 380 to 382; paired 95% CI [+0.41, +2.88]), +2.47 pp with the descriptors without class names ([+0.48, +4.48]), and +2.47 pp on the extended SiFC of 1,056 images ([+0.49, +4.39]).

We kindly request the reviewer to note that these gains are measured over the global-only variant, which is also our setting (Table 1, line 362), built on our descriptors. It already gains +8.22 pp over the same MLLM with class names only (line 359), +16.46 pp over CLIP zero-shot (line 353) and +12.20 pp over CBD (line 354), all training-free.

In short, the additional calls add localization and a modest but consistent gain, on top of a global-only variant that is also part of our contribution.

We are grateful to the reviewer for these comments, which helped us see where the paper was not clear enough. Following them, we have updated the title, added the concept localization results to the appendix, and made the difference from zoom-in methods clearer. We hope these answers address the concerns, and we would be glad to clarify anything further.

Thank you for the detailed rebuttal and additional experiments. I have a few remaining questions:

  1. The new localization evaluation covers only 185 images and one key concept per class. How representative are these results of the full concept set? Also, does a smaller distance to the concept center necessarily imply more faithful concept-level explanations?

  2. The response clarifies how home-scale routing differs from existing zoom-in methods in implementation, but I am still unclear about its methodological novelty. Beyond using concept-specific scales and explicit concept scores, what is the key technical distinction from existing multiscale visual search approaches?

  3. The reported accuracy improvement is modest, while the additional efficiency analysis indicates approximately 4.8× inference latency compared with global-only. Could the authors clarify whether the localization benefits justify this overhead, particularly relative to simpler crop-based alternatives?

Thank you again for the clarifications.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.