- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents APERTURE, a training-free concept-based recognition framework for fine-grained remote sensing imagery. The method replaces CLIP-style concept matching with MLLM-based binary concept verification, derives continuous concept evidence from Yes/No decoding probabilities, and introduces concept-specific home scales with greedy quadtree routing to recover small visual components. The paper further introduces SiFC, a concept-centric benchmark covering six industrial facility classes across three countries.
Critical strengths
-
The problem is important and well motivated for remote sensing, where fine-grained components span substantially different physical scales.
-
The use of an MLLM as a training-free concept verifier is simple and practically meaningful, and the use of Yes/No decoding logits is preferable to verbalized confidence.
-
SiFC is a potentially valuable benchmark for interpretable fine-grained remote-sensing recognition.
Critical weaknesses
-
Limited gains from the multiscale routing strategy. The empirical benefit of the proposed multiscale routing mechanism appears relatively modest. On SiFC, the global CCCD setting already achieves 78.62 macro-F1, while the full APERTURE framework reaches 80.19. Moreover, routed home-scale evaluation performs almost identically to the best fixed-crop baseline. These results suggest that the routing module contributes only marginally to the overall performance improvement.
-
Insufficient disentanglement of individual contributions. The current experiments modify several factors simultaneously, including the use of an MLLM, structured descriptors, country conditioning, and multiscale routing. As a result, it is difficult to determine which components are primarily responsible for the observed gains. A more systematic factorial ablation would help isolate the contribution of each design choice.
-
Lack of computational cost analysis. The proposed framework requires repeated inference with a large MLLM over multiple concepts and image regions, which may introduce substantial computational overhead. However, the paper does not report practical efficiency metrics such as inference latency, the number of MLLM calls per image, GPU usage, or computational cost. Such analysis is important for assessing the practicality of the method.
-
Limited dataset scale and statistical robustness. SiFC contains only 800 images across six classes, with noticeable imbalance across classes and countries. Given that some of the reported improvements are on the order of only 1–2 macro-F1 points, it is unclear whether these gains are statistically robust. Reporting confidence intervals or significance tests would strengthen the empirical conclusions.
Questions for authors
Some inference prompts appear to explicitly include the class name. Could the authors clarify whether this introduces class-level priors into the concept scores, rather than making them depend solely on the corresponding visual concepts? If so, how does this affect the interpretation of APERTURE as a concept bottleneck model?
5 comments
Response to Reviewer FcK5: Rebuttal part 2 of 3
W2. Insufficient disentanglement of individual contributions. The current experiments modify several factors simultaneously, including the use of an MLLM, structured descriptors, country conditioning, and multiscale routing. As a result, it is difficult to determine which components are primarily responsible for the observed gains. A more systematic factorial ablation would help isolate the contribution of each design choice.
We thank the reviewer for this suggestion. Following it, we ran a factorial ablation on SiFC. The table below gives the result, where each row is one combination of the design choices.
We thank the reviewer for this suggestion. Following it, we ran a factorial ablation on SiFC. The table below gives the result, where each row is one combination of the design choices.
| # | Setting | MLLM | Concept descriptors | Class conditioning | Country conditioning | Multiscale routing | Macro-F1 |
|---|---|---|---|---|---|---|---|
| 1 | CLIP ViT-L/14, class name only | ✗ | ✗ | ✗ | ✗ | ✗ | 62.16 |
| 2 | CLIP ViT-L/14, general descriptors (GCD), global-only | ✗ | ✓ | ✗ | ✗ | ✗ | 41.10 |
| 3 | CLIP ViT-L/14, general descriptors (GCD), with routing | ✗ | ✓ | ✗ | ✗ | ✓ | 46.52 |
| 4 | CLIP ViT-L/14, class descriptors, global-only | ✗ | ✓ | ✓ | ✗ | ✗ | 42.11 |
| 5 | CLIP ViT-L/14, class descriptors, with routing | ✗ | ✓ | ✓ | ✗ | ✓ | 44.29 |
| 6 | CLIP ViT-L/14, class and country descriptors (CCCD), global-only | ✗ | ✓ | ✓ | ✓ | ✗ | 44.99 |
| 7 | Gemma 4 31B, class name only | ✓ | ✗ | ✗ | ✗ | ✗ | 70.40 |
| 8 | Gemma 4 31B, general descriptors (GCD), global-only | ✓ | ✓ | ✗ | ✗ | ✗ | 70.78 |
| 9 | Gemma 4 31B, class descriptors, global-only | ✓ | ✓ | ✓ | ✗ | ✗ | 72.37 |
| 10 | Gemma 4 31B, class and country descriptors (CCCD), global-only | ✓ | ✓ | ✓ | ✓ | ✗ | 75.67 |
| 11 | Gemma 4 31B, class descriptors, with routing | ✓ | ✓ | ✓ | ✗ | ✓ | 75.62 |
| 12 | Aperture | ✓ | ✓ | ✓ | ✓ | ✓ | 78.14 |
Gain (pp) from using the MLLM in place of CLIP, at the same setting. The numbers in brackets are the two rows compared.
| Setting | Gain (pp) |
|---|---|
| Class name only | +8.24 (1 → 7) |
| General descriptors (GCD), global-only | +29.68 (2 → 8) |
| Class descriptors, global-only | +30.26 (4 → 9) |
| Class descriptors, with routing | +31.33 (5 → 11) |
| Class and country descriptors (CCCD), global-only | +30.68 (6 → 10) |
Gain (pp) from each other design choice, with each backbone.
| Design choice | With CLIP ViT-L/14 | With Gemma 4 31B |
|---|---|---|
| Concept descriptors | −21.06 (1 → 2) | +0.38 (7 → 8) |
| Class conditioning | +1.01 (2 → 4), −2.23 (3 → 5) | +1.59 (8 → 9) |
| Country conditioning | +2.88 (4 → 6) | +3.30 (9 → 10), +2.52 (11 → 12) |
| Multiscale routing | +5.42 (2 → 3), +2.18 (4 → 5) | +3.25 (9 → 11), +2.47 (10 → 12) |
We can observe that the MLLM is the largest factor, with about +30 pp in every setting that uses descriptors and +8.24 pp with class names only. Country conditioning and multiscale routing add smaller gains that are positive in every comparison, +2.52 to +3.30 pp and +2.18 to +5.42 pp. Concept descriptors and class conditioning give small or mixed gains on their own.
In summary, the MLLM brings most of the gain, and country conditioning and multiscale routing each add a smaller but consistent gain on top.
Q1. Some inference prompts appear to explicitly include the class name. Could the authors clarify whether this introduces class-level priors into the concept scores, rather than making them depend solely on the corresponding visual concepts? If so, how does this affect the interpretation of APERTURE as a concept bottleneck model?
We thank the reviewer for pointing this out, and we apologize for the mistake on our side.
The class names should not appear in these prompts, since they can bring class-level priors into the concept scores. We removed them from the verification and routing prompts and ran the experiment again. As the table below shows, Aperture still improves over global-only.
| Descriptor | Gain (pp) | Paired 95% CI (pp) |
|---|---|---|
| HSR-CCCD, with class names (paper) | +1.57 | [+0.41, +2.88] |
| HSR-CCCD, no class names (new) | +2.47 | [+0.48, +4.48] |
In short, the gain remains without class names, which suggests that the concept scores rely mainly on the visual concepts, so Aperture can still be read as a concept bottleneck model.
Response to Reviewer FcK5: Rebuttal part 3 of 3
W4. Limited dataset scale and statistical robustness. SiFC contains only 800 images across six classes, with noticeable imbalance across classes and countries. Given that some of the reported improvements are on the order of only 1–2 macro-F1 points, it is unclear whether these gains are statistically robust. Reporting confidence intervals or significance tests would strengthen the empirical conclusions.
SiFC is a small dataset and is not fully balanced across classes and countries. We want to clarify that SiFC focuses on fine-grained classes that are recognized through their components (lines 157 to 159), and such facilities are few. The imbalance is also present in the ground truth, where some facility types are much rarer than others. When we tried to extend SiFC, we could add only 256 images from the three countries. For example, we could add only 4 nuclear power plants and 8 LNG terminals. This suggests that a large and balanced fine-grained dataset is hard to build.
| Class | Original | Added | Now |
|---|---|---|---|
| Iron and steel plant | 180 | 49 | 229 |
| LNG terminal | 49 | 8 | 57 |
| Nuclear power plant | 74 | 4 | 78 |
| Oil refinery | 159 | 9 | 168 |
| Power plant | 180 | 138 | 318 |
| Sewage treatment plant | 158 | 48 | 206 |
| Total | 800 | 256 | 1,056 |
The table above shows what happened when we tried to extend SiFC. We could add only 256 images from the three countries, for example only 4 nuclear power plants and 8 LNG terminals, which suggests that a large and balanced fine-grained dataset is hard to build. Besides SiFC, the paper also has a facility subset of fMoW, with 400 images of five other facility classes from the USA, France and Russia (Table 8, line 1136), where Aperture reaches 76.55% macro-F1 (line 366).
Because SiFC is small, we also checked whether the gains are statistically robust, as the reviewer suggested. We computed paired 95% confidence intervals with a bootstrap over the 800 images (10,000 resamples, stratified by country and class, so every resample keeps the same imbalance). The interval lies above zero in every comparison of the table below.
| Comparison | Diff. (pp) | Paired 95% CI (pp) |
|---|---|---|
| Aperture − global-only, descriptors as in the paper | +1.57 | [+0.41, +2.88] |
| Aperture − global-only (No class Name) | +2.47 | [+0.48, +4.48] |
| Aperture − supervised CBM head, leave-one-country-out | +12.35 | [+8.85, +15.86] |
| Aperture − CLIP ViT-L/14, class name only | +15.98 | [+12.13, +19.97] |
| Aperture − CLIP ViT-L/14, CbD | +11.72 | [+7.96, +15.57] |
| Aperture − RemoteCLIP ViT-L/14, CbD | +38.81 | [+35.00, +42.76] |
| Aperture − GeoRSCLIP ViT-L/14, CbD | +19.93 | [+16.32, +23.69] |
| Aperture − EarthDial 4B RGB, class name only | +23.94 | [+20.13, +27.89] |
| Aperture − EarthDial 4B RGB, CbD | +34.68 | [+30.58, +38.79] |
To sum up, every interval lies above zero, which suggests that the gains, including the small gain from multiscale routing, are reliable at this dataset size.
We thank the reviewer again for the comments. Based on them, we ran the factorial ablation, measured the computational cost, and computed confidence intervals for the gains. We are happy to answer any further questions.
Response to Reviewer FcK5: Rebuttal part 1 of 3
We thank the reviewer for the detailed comments, and we are glad the reviewer values the training-free concept verifier and the Yes/No decoding logits. We answer each point below.
We agree with the reviewer that the gain in macro-F1 from routing is modest. What routing mainly adds is concept localization. Global-only scores each concept on the whole image, so it tells whether the concept is present but gives no location. Aperture keeps this global answer and also scores each concept at its home scale, so it also tells where the concept is. We checked this for one key concept per class on 185 images. Full details are provided in Appendix Table [7,8].
We can observe that Aperture reports the key concept in more images than global-only, 88.6% against 74.6%. It also gives a location for it in 74.1% of the images, while global-only gives none.
In comparison with the fixed-crop baseline, the accuracy is close. To see what routing adds beyond accuracy, we measured how near each view is to the concept. We marked the position (x, y) of the key concept in the images, and measured the Euclidean distance from this position to the centre of the full image, of the best-scoring fixed 2048 × 2048 crop and of the routed home-scale crop, on the images where all three report the concept.
We can observe that the center of the full image is closer to the concept than the fixed 2048 × 2048 crop, because the images are centered on the facilities. Nonetheless, the routed home-scale crop of Aperture is the closest of the three. So although the accuracy is close to a fixed crop, routing brings the view nearer to the concept.
In summary, while the gain in macro-F1 is modest, multiscale routing also shows where each concept is, and its view is closer to the concept than the full image or a fixed crop.
We thank the reviewer for raising this, and we agree that the paper should report these costs. We have now measured them for global-only, for Aperture and for fixed grids of crops. The table below gives the number of MLLM requests per image, the time per image and the GPU-hours, where one request is one MLLM call with one image and one question.
Setup. Gemma 4 31B on one NVIDIA A100 80 GB GPU per worker, BF16 precision, 224 × 224 image inputs, and a vLLM GPU-memory utilization target of 90%. The model fits on one GPU, without tensor parallelism.
As the table shows, Aperture needs about 4× the requests and 4.8× the time of global-only, so it does cost more. The other way to read small concepts is a fixed grid of crops, and this costs more at every size, from about 5× the requests of global-only for its cheapest form to about 257× for a grid at the 256 px scale. This growth matches the time complexity stated in the paper (lines 318 to 319), where an exhaustive search needs O(4ℓ) MLLM calls for ℓ zoom levels and greedy routing needs O(ℓ+1).
Overall, Aperture costs more than global-only, and routing keeps this cost below any fixed grid of crops.