acceptodds
Under review as a conference paper at ICLR 2027

What Survives Extreme Compression? Vision-Language Embedding Alignment at Microcontroller Scale

Abstract

We ask what remains of CLIP’s image - text embedding geometry when ViT-B/32’s 87.8 M-parameter image tower is distilled into a 724 K-parameter microcontroller encoder whose class prototypes the teacher’s text encoder computes offline. Distilling on CC3M, we find a sharp capacity - granularity boundary: the student keeps 39% of teacher accuracy on COCO with classes weighted equally, 12% on ImageNetV2, 5 - 9% on fine-grained sets and near chance on two robotics sets, a boundary that tracks pretraining coverage. It spans ≈93 effective dimensions where the teacher needs ≈277, and a stronger teacher adds up to 3.2 points at the deployed dimension at no device cost. One coordinate of the teacher is its prototypes’ shared direction: CLIP ViT-B/32’s coordinate 133 carries 79% of the mean prototype’s squared norm and enters the prefix only at 256 dimensions, where it adds ≈6 per-image COCO points, all on “person”, while class-balanced accuracy falls; projecting it out lifts class-balanced 256d above 128d on COCO and ImageNetV2. Alignment is also teacher -specific: prototypes from any other text encoder score at chance, but a closed-form map fitted on class names alone recovers 64-71% of accuracy offline. The INT8 student runs at 2.5 FPS on an ESP32-S3 and fits an STM32H7 in 1,085 KB flash (464 KB arena, emulated); its 128 px input costs eleven points of top-1 at ten COCO classes (37.9% vs. 48.8%).

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
vision-language models, knowledge distillation, model compression, Matryoshka representation learning, embedding geometry, rogue dimensions, zero-shot classification, on-device inference, anisotropy / rogue dimensions, TinyML, microcontrollers

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_G2fEOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper distills CLIP ViT image modual into a 724K-parameter MobileNetV2-0.35× student using Matryoshka contrastive and logit distillation on CC3M. The results show that a capacity–granularity boundary about 93 vs 277 effective dimensions, a dominant teacher coordinate (133) that distorts per-image COCO accuracy at 256-d, teacher-specific alignment while other encoders' prototypes score at chance, partly recoverable by a closed-form map, and INT8 deployment on an ESP32-S3 (measured) and STM32H7.

Critical strengths

This work is practival impelementation oriented, and has real hardware measurements and good reproducibility details. The experiment results are explained clearly and easy to follow. The method is problem-based and engineering-based.

Critical weaknesses

It seems the causal claims are weakly supported by the experiment results because capacity and pretraining coverage are not separated with one student size and one data scale setting. Also the deployed 128 px INT8 pipeline reaches 16.0% per-image COCO accuracy, below the 31.6% majority class, and main results use a 224 px FP32 student that cannot run on the MCU. The STM32H7 is only emulated and no latency is reported, and the 128 px loss was not mitigated by training at that resolution.

Questions for authors

I would like to aks how does the conclusion about coordinate 133 / mean-direction removal change with multiple seeds and a metric that is not majority-dominated?

Reviewer_yKdcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper distills a compact image encoder into CLIP's text-embedding space and studies accuracy across embedding dimensions, datasets, prototype sources, and microcontroller settings. It finds that performance depends strongly on class granularity and vocabulary size, analyzes a dominant CLIP prototype direction, and measures 2.5 FPS on ESP32-S3.

Critical strengths

The evaluation distinguishes class-balanced from per-image accuracy, includes several teacher models and embedding dimensions, and is unusually clear about deployment limitations. The prototype-portability study makes the work more than a model-size demonstration.

Critical weaknesses
  • The adaptation of foreign text prototypes is fitted on 983 ImageNet class names, while the paper acknowledges that ImageNet hyponyms of held-out COCO classes remain in this lexical training set. Since the recovery claim is a key result about portability, the semantic overlap makes the reported 64–71% recovery difficult to interpret as transfer to unseen concepts.

  • At the deployed 128-pixel input, accuracy falls substantially relative to the 224-pixel evaluation, and results on the robotics datasets are near chance. The paper acknowledges this, so the contribution is best understood as a small-vocabulary, coarse-category operating point rather than general-purpose MCU vision-language recognition.

Questions for authors
  1. For the 983-name prototype-adaptation fit, is there an existing count of ImageNet labels that are hyponyms or near-synonyms of the held-out COCO classes? This would help determine how much semantic overlap may explain the reported recovery.
  2. Has the final firmware graph been run to completion on the physical STM32H7, or is the STM32H7 claim limited to flash size and emulated arena usage? Please distinguish this from the ESP32-S3 result, which uses external PSRAM.
1 comment

I thank the authors for the clarifications, which resolve my question on the on-board measurements. However, the overlap analysis confirms that the prototype-portability result reflects concepts that are lexically held out but semantically present, and the semantically disjoint split needed to support a transfer claim has not been reported. Since the authors also agree that the contribution is limited to a small-vocabulary, coarse-category operating point, I maintain my original score.

Reviewer_SQRgOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper analyses how much of CLIP performance survives a distillation into a microcontroller-scale image encoder. The resulting performance, with their methodology, reaches 39% of the teacher’s class-balanced COCO accuracy, 5-9% on other (fine-grained) datasets, and performs near random on robotics datasets.

Critical strengths

Repeated experiments for accounting variation

Critical weaknesses
  • The paper seems to be heavily AI-written, leading to some conclusions being formulated stronger than the reported numbers in the experiments support. For example, the caption of Figure 8 claims that higher-dimensional embeddings dominate at every rank and establish an effective hierarchy for retrieval. However, Table 23 shows that text-to-image retrieval deteriorates from 128 to 256 dimensions across all reported metrics. Combined with the overall poor performance of their distillation approach, it raises the question if the experiments were even human-reviewed.
  • With respect to the last comment, the paper is also very hard to read in some places as it includes the typical Claude formulations for paper writing by over-representing quantitative numbers in the main text.
  • The practical significance of the contribution remains limited. Although the paper demonstrates MCU deployment and investigates several corrective interventions, the resulting system remains near chance on the robotics benchmarks and performs poorly on fine-grained recognition.
  • Although the paper includes a pseudo-label supervised baseline, it lacks a comparison with a straightforward feature-distillation method using the same student architecture, training data, and deployment budget. Unlike the supervised baseline, such a method could also preserve compatibility with text prototypes and support vocabulary changes. This comparison is particularly important because the paper interprets poor performance as evidence of limits under extreme compression.
  • While the goal is the microcontroller-scale, the capacity analysis is too limited to establish the main claims of the paper. Although the paper compares three small backbone widths, it does not trace performance from larger, less compressed students down to the microcontroller regime. Consequently, the experiments do not establish whether fine-grained recognition deteriorates gradually or exhibits a distinct capacity threshold. A broader capacity sweep, with training data and objectives held fixed and per-dataset performance reported, would provide stronger support for this central interpretation.
Questions for authors
  • How does the same student compare with direct image-feature distillation, using matched data, compute, output dimension, and deployment budget? Table 6 indicates that logit distillation adds little once the Matryoshka objective is used.
  • Section 4.7 evaluates the ten added classes only against their ten prototypes. What are old-class and new-class accuracies when all 80 prototypes compete after the update?
  • Do you have results from training or fine-tuning at 128×128, and class-balanced results for the resulting INT8 pipeline across the evaluated datasets?
  • Do you have matched results after applying a shared orthogonal rotation, or a projection fitted only on training data, to the teacher image/text representations before student training? Would the reported dimension and teacher-size trends remain?
  • Do you have results for larger students trained with the same data and objective that show how fine-grained recognition changes as model capacity decreases toward the microcontroller regime?
  • The paper explicitly acknowledges that resolution-matched training and patch-based inference were not tried. Is there any reason why not because in the referenced works, this seems to be a very important component in making distillation work in tiny model regimes.

Comments on the paper

1 comment

Dear reviewers,

It seems that the authors have carefully responded to your reviews.

If you haven't already, please take your time to go over them and let authors know what you think, to your earliest convenience.

Best,
AC.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.