acceptodds
Under review as a conference paper at ICLR 2027

Structure over Pixels: Learning Variable-Length Visual Programs

Abstract

Discrete visual tokenizers map images to ordered sequences of tokens, providing a natural representation for structural scene descriptions. Most use a fixed sequence length, while adaptive methods often require post-hoc search or choose among a small set of rates that control the length. We propose STROP, a discrete tokenizer that learns both a visual program and its image-dependent active length. A length head is trained with a four-phase curriculum using local rate–distortion probes against frozen DINOv3 features, then predicts the active prefix in a single forward pass. At a matched rate of about nominal bits per crop, the adaptive model improves segmentation over a separately trained fixed-length baseline on four benchmarks (by – mIoU), and it also beats a fixed baseline that uses more bits. STROP programs also yield higher segmentation mIoU than FlexTok, One-D-Piece, and ALIT at similar or higher rates, under the same readout architecture and training protocol. STROP therefore learns useful per-image sequence lengths without post-hoc search or a predefined set of compression rates.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
variable-length tokenization, structural scene representation, symbolic representations, self-supervised distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_dbadOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes STROP, a discrete visual tokenizer that learns image-dependent token sequence lengths. It combines a Transformer-based program generator, vector quantization, a feature interpreter, and a lightweight length-prediction head. The core idea is to train the length head using local rate–distortion signals measured against frozen DINOv3 features, enabling the model to select an appropriate token prefix in a single forward pass without inference-time search or a predefined set of compression rates. STROP presents a technically plausible and empirically promising approach to learning image-dependent discrete visual representations. The rate-matched experiments provide convincing evidence that adaptive token allocation can improve downstream segmentation performance over fixed-length representations under the evaluated setup.

Critical strengths
  1. Relevant problem and clear motivation. Allocating representation capacity according to image complexity is a meaningful direction for visual tokenization. The paper appropriately focuses on preserving task-relevant features rather than optimizing pixel reconstruction alone.

  2. Controlled evaluation against fixed-length tokenizers. The rate-matched experiments compare STROP with separately trained fixed-length models under a shared teacher, architecture, and training setup. STROP improves segmentation mIoU by approximately 1.6–3.1 points at a similar nominal bit budget across four benchmarks. The use of multiple tokenizer and probe seeds strengthens this evidence.

  3. Broad empirical and analytical evaluation. The paper includes component ablations, multiple downstream tasks, token-erasure analyses, and experiments investigating the relationship between program length and scene complexity. The authors also discuss several limitations and provide implementation code.

Critical weaknesses

1. Limited methodological originality. Flexible-length visual tokenization has been explored by FlexTok, One-D-Piece, ALIT, and other adaptive methods, while KARL also predicts image-dependent token lengths in a single forward pass. STROP’s main distinction is its local rate–distortion supervision in frozen teacher-feature space. Still, the lack of a direct comparison with KARL or other controlled alternatives makes the advantage of this design insufficiently clear.

2. Limited comparability of adaptive-tokenizer baselines. Comparisons with FlexTok, One-D-Piece, and ALIT use a shared readout but involve different training data, teachers, and objectives, making it difficult to attribute performance gains specifically to adaptive length selection. Controlled comparisons under matched training and evaluation settings would better isolate the contribution of STROP’s proposed mechanism.

3. Insufficient evidence of computational efficiency. The paper primarily evaluates efficiency through nominal bit rate and token count, which do not necessarily translate into lower end-to-end computation. Since STROP generates a fixed maximum-length candidate sequence before truncation, reporting inference latency or FLOPs would clarify whether it reduces actual computational cost.

4. Limited evidence for semantic interpretability. Token-erasure analyses and correlations between program length and scene complexity suggest that STROP captures meaningful visual structure, but they do not establish that individual codes correspond to interpretable concepts or form compositional visual programs. The reported difficulty of reliably decoding object attributes further limits the claim that these representations provide a symbolic interface for downstream reasoning.

Questions for authors
  1. It is necessary to provide a controlled comparison with KARL or otherwise isolate the benefits of local rate–distortion supervision over existing adaptive-length prediction mechanisms.

  2. How much of STROP's improvement over public adaptive tokenizers comes from adaptive length selection, as opposed to the DINOv3 feature-distillation objective? Can this be tested under a shared teacher, training dataset, and readout protocol?

  3. Does STROP reduce the actual computation required to generate visual tokens, or primarily reduce the number of tokens passed to the interpreter and downstream models? What are the corresponding latency or FLOPs measurements?

  4. What evidence supports interpreting the learned codes as compositional visual programs, rather than compressed latent representations with localized effects? How reliably can individual tokens or token combinations be associated with semantic objects and attributes?

3 comments

Thank you for the detailed review. We do not claim single-pass length prediction to be new; we differ from KARL in how the length head is supervised and in needing no quality target at inference. On compute, STROP mainly shortens the representation it passes on, and generating the shorter program saves only a modest amount; we also narrow "program" to what the evidence shows. More details follow.

Limited methodological originality

KARL predicts halting in a single pass that is conditioned on a reconstruction-quality target supplied at inference. STROP's length head is trained inside a prefix curriculum from teacher-feature reconstruction and needs no target at inference.

We have not run a KARL control aligned with our configurations. A fair one would involve training KARL's halting on DINOv3 features and calibrating its quality target to our rate, which would change its published recipe. However, we did run the closest controlled alternative, i.e., supervision from reconstruction error alone (see the reply to DXCD, section "Length supervision and the training budget"): the probe factor sets the operating point, and a fixed scale at the converged value is a valid, simpler variant.

Limited comparability of adaptive-tokenizer baselines

We expect most of the gap w.r.t. the public tokenizers originates in the feature-reconstruction objective and the teacher, and our discussion of Table 2 does not attribute it to adaptive length. Our reply to ijYv ("Comparability of public tokenizers") discusses the confounders and evidence. Adaptive length is tested inside STROP under a shared teacher, data and readout in two ways: against separately trained fixed-length models (Table 1), and by randomly reassigning predicted lengths across images while keeping the representation and the exact token total fixed (reply to DXCD, "Image-specific allocation").

Insufficient evidence of computational efficiency

STROP mainly reduces the number of tokens it passes on; the computation needed to generate them falls only modestly. The length head reads pooled encoder features, so the length is known before the program is generated. Because the generator is causal, generating only the first L tokens gives the same tokens as generating all 64 and truncating. The released implementation generates 64 tokens and then truncates; generating only the prefix needs no retraining, only a different inference loop. Measured on an H100 (DINO-S, 256-pixel crops, from GPU-resident input to interpreted features), prefix generation lowers the expected compute from 22.4 to 21.0 GFLOP per crop, or from 9.9 to 8.5 GFLOP (14%) for the STROP part after DINO. Expected latency drops by only 0.3–1.0% (FP32 and BF16, batch sizes 1 and 16, with fixed-length timings weighted by the observed length distribution).

The saving is small for two reasons: DINO's 12.5 GFLOP do not depend on length, and the generator processes the 256 patch tokens together with the program positions, so a 24-token program shortens its input sequence by only about 12%. The larger change is in the output. At about 250 bits per crop, STROP passes on 24 codes on average instead of 64 (640 bits), 2.6 times fewer. This reduces storage and the input length of any downstream model whose cost grows with the number of tokens it reads. A dense readout such as our segmentation probe gains little time, because its 256 output positions dominate its cost. We will add these measurements to the final version.

Limited evidence for semantic interpretability

By "program" we mean a discrete, ordered description executed by a learned interpreter. We will state this definition in Section 1 and avoid wording that suggests symbolic composition we have not shown. What the evidence does show:

  • Localized effects. Erasing one token changes a localized part of the interpreted field (Figure 1); token pairs show localized interactions (Figure 11 in the appendix); clustering the effects of erasures without labels gives regions similar to DINO's (Figure 4).
  • Object alignment. On CLEVR, the most influential tokens tend to correspond to individual objects (Figure 16), and program length correlates with the number of objects in the scene (r≈0.74r \approx 0.74).
  • Meaningful order. Object count and attribute readouts on CLEVR improve when token positions are given, which is an advantage shuffling removes (Table 20). On COCO, shuffling the positions lowers teacher cosine from 0.86 to 0.53 (Table 9).

At the current stage of STROP advancement, individual codes cannot yet be reliably related to visual semantics. Color and material are hard to read on CLEVR (Table 20), and frozen LLMs did not beat a dataset prior on attributes. The representation is spatially organized and ordered, with parts that can be located and manipulated. Forming codes with interpretable semantic meanings reused across scenes remains a further goal, and we will state this limit more plainly in the revised paper.

Thank you for the detailed response. My major concerns have been addressed, and I will maintain my original positive decision.

Thank you for your time and constructive feedback throughout the review and discussion period. We are glad our response addressed your main concerns. The revision will include what we committed to in our reply:

• the distinction from KARL in how the length head is supervised and in needing no quality target at inference;

• the confounders of the comparison with public tokenizers in Table 2;

• the compute and latency measurements of prefix generation;

• a definition of "program", with the limits of the semantic evidence stated plainly.

Since our previous reply, we have also measured how much the length head loses by allocating from per-crop reconstruction error rather than with an oracle at the same token budget; the results are in our latest reply to DXCD and will be added to the paper.

Reviewer_ijYvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a framework that learns to represent images with small set of discrete tokens that is (importantly) adaptively chosen by based on the image. A generator transform summarizes visual tokens in a structured vocabulary and an interpreter reads it back to reconstruct DINOv3 features, making the pipeline self-supervised. The representations are still useful for semantic tasks, outperforming related approaches and baselines.

Critical strengths

The core idea of the paper of having an adaptive length tokenizer is a conceptually interesting approach, it is presented convincingly, and supported through experimental evidence.

Critical weaknesses

The works specifically motivates a sequential auto-regressive organization, which is a notable design choice for vision. It is interesting to see that spatial configuration outperforms the proposed strategy. It would be helpful if the authors could discuss this comparison (and concrete setups etc.), maybe potentially also show this comparison also on other tasks (not just semantic segmentation). It would help to understand whether a simple baseline (it is a simple baseline, right?) is actually more effective than the proposed strategy.

I would not treat the questions in the question section as critical weaknesses but the answers would influence whether I would see this as a weakness.

Questions for authors

The downstream task evaluation is limited to conventional tasks. Could the authors comment on whether this representation / vocabulary could be also useful for other applications, eg, generation or potentially even reasoning in visual space or so? This could potentially substantially increase the impact of the work. Currently, the comparison to previous models is not directly comparable due to rather different settings (in how DINO features are used). Are the other ways to make those models comparable? If more evaluations are beyond what is feasible, I would appreciate a discussion of the confounders in the comparison.

3 comments

Thank you for the questions. At a fixed length, the inclusion of grid's location prior increases the mIoU by about 2 points, and the ordered layout is not better in itself. Its role is to make adaptive length possible – and adaptive STROP closes that gap on the segmentation task.

Spatial grids and ordered programs

The spatial grid baseline is simple in its bottleneck layout, but trained like STROP: a learned compressor maps DINO features to a 5x5 grid, quantized with the same codebook and decoded by the same interpreter. Each code is tied to one grid cell, a strong prior for dense prediction.

  • At 25 codes, the sequential model trails the baseline (grid variant) by 2.4 and 2.2 mIoU (decoded-feature and program readouts, four-benchmark mean; Table 8). Adaptive STROP at the same average rate matches the grid variant (Table 3).
  • The order carries information. Shuffling a trained program lowers the teacher cosine similarity of interpreted features (to DINO embeddings) from 0.86 to 0.53 (Table 9); retraining without the causal mask costs 2.5 mIoU on average (Table 3).
  • A code does not need to follow a grid cell: a grid splits an object across cells, a program code can cover it whole. On CLEVR, the most influential tokens overlap individual objects, increasingly as tokens are added (Figure 16). On COCO, grouping patches by which tokens affect them (k-means on per-patch token-erasure effects, no labels) gives coherent regions (Figure 4). This is our main motivation for ordered programs, though not yet compared directly with the grid.

Please note that adaptive length needs an order in which to drop codes. A prefix is a shorter description of the whole image; dropping grid cells removes regions.

We also ran the comparison on NYUv2 depth at 25 codes (same data, readout capacity and training budget; single seed). The grid variant is slightly better from decoded features (RMSE 0.54 vs. 0.55 meters), and the program readouts tie at 0.54. Adaptive STROP reaches 0.56 from both readouts, but its readout head differs slightly, so this is not a strict comparison. On dense geometry the location prior helps, and adaptive length does not recover it; the gap is small.

Applications and connection to language models

A STROP program is a short sequence of discrete codes, the same form as language-model input. In our experiments, a scene costs about 24 codes on average, compared to 256 teacher patch tokens, and the count follows the complexity of the scene content. Any prefix is valid, so visual context can be shortened per query. A grid can also be flattened into a sequence, but its order is imposed after training. On the contrary, STROP’s order is the one its generator and interpreter were trained with.

Appendix M tests the simplest form of the interface: frozen LLMs, 50 in-context examples, no training. On 15 held-out CLEVR scenes, the two large models cut object-count error from 2.27 objects for a naive baseline (the average scene of the in-context examples, ignoring the codes) to 0.93–1.53 at every codebook size, and also improved position error; attribute accuracy did not exceed the baseline’s. Connecting programs to language models is our main follow-up line.

For generation in general, an autoregressive prior over programs, decoded by the interpreter, would produce programs whose length depends on the content, without a separate length predictor.

Comparability of public tokenizers

Table 2 applies the same pipeline to all tokenizers (STROP also counts its 6-bit length field). Several differences remain:

  1. Teacher and objective. STROP reconstructs DINO features, which are themselves strong for segmentation, while the public tokenizers reconstruct images. This likely favors STROP more than any other difference (points 2-4 below). Our own ablation points the same way: when pixel-reconstruction gradients reach the STROP generator, mIoU drops by 7.9 and 8.6 for the latent and program readouts (Table 3).
  2. Training data. Table 2’s STROP is trained on COCO, while FlexTok and ALIT are trained on ImageNet-1k. When trained on ImageNet, STROP performs segmentation as well as the COCO-trained model (Table 16), keeps the adaptive-vs-fixed gain (Table 9), and still exceeds FlexTok at every budget we evaluated. Therefore, the advantage over public tokenizers does not come from training on COCO.
  3. Rate. Every public tokenizer in Table 2 is at or above 250 bits, which favors them, yet STROP at 160 bits still beats FlexTok at 256.
  4. Model size and codebook. We cannot tell which configurations these favor without retraining.

Table 2 thus measures how much segmentation information each model keeps per bit; we do not attribute the gap to adaptive length, whose claims rest on the fixed-length and grid baselines and the permutation control (see also our reply to DXCD, section "Image-specific allocation").

We will expand the confounder discussion under Table 2 accordingly.

Thank you for the rebuttal. Generally, I keep a weak positive rating for the paper but will also consider the final response to reviewer DXCD. In any case, I would encourage the authors to clearly communicate what parts resulted in the performance gains, particularly also related to Tab. 2. All the best.

Thank you. Following your suggestion, this is how the revision will attribute the gains.

Table 2, STROP against public tokenizers. Within STROP, allowing pixel-reconstruction gradients into the generator costs 7.9 latent-readout and 8.6 program-readout mIoU points (Table 3). This makes the training objective a plausible explanation for the gap to public tokenizers, but Table 2 does not isolate its share. Training on ImageNet instead of COCO does not change the comparison (Table 16), and the rates favor the public tokenizers. Table 2 therefore measures segmentation information per bit, and we do not attribute its gap to adaptive length.

Table 1, STROP against fixed K=25. Image-specific allocation accounts for +0.94 program-readout mIoU points on VOC and +0.28 on COCO-Stuff27 (near-uniform lengths at the same token budget, mean over three tokenizer seeds) and for no measurable part on ADE20K and Cityscapes. Training on variable-length prefixes, which the fixed-length baselines lack, is a plausible explanation for the remainder. The probe factor m sets the overall rate and does not set the length of individual images (see our reply to DXCD). We will add this breakdown to Section 4.

Language models. On CLEVR, program length correlates with object count (r ≈ 0.74), so we checked how much of the Appendix M count result length alone explains. On 100 new CLEVR scenes (codebook size 64, 50 in-context examples, no training), DeepSeek-V3.1 received the scene's own program, a program of the same length from another scene, or the length alone. All three beat the naive baseline, the average scene of the in-context examples (object-count error 2.01). The object-count error was 1.24 with the scene's own program, 1.33 with another scene's program and 1.44 with the length alone (all 100 scenes; 98 length-only answers could be parsed). The point estimates favor the scene's own program, but the paired 95% intervals of both differences include zero, and the same holds for localization. On these 100 scenes we detected no extra count or localization accuracy from supplying the correct codes, so we will revise the Appendix M sentence so that it does not attribute the zero-shot improvement to the codes themselves.

The codes do carry this information when a linear readout is trained on them. In Table 20, linear readouts trained on CLEVR programs predict object count. For the fixed-length model (K=32, codebook size 64), every program has the same length, and the object-count error is 0.61. For the adaptive model with codebook size 64, a readout trained on the length alone reaches 1.29, in the same range as the frozen LLM, while the codes with their positions reach 0.74. Reading the codes therefore needs training, which a frozen LLM with in-context examples does not get. We will add these controls to Appendix M and revise its text accordingly.

Reviewer_DXCDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Like several earlier works (FlexTok, ElasticTok, and KARL), the authors argue that adapting the number of visual tokens to image complexity can improve representation quality at a given average token budget. For this, the authors propose a discrete visual tokenizer, STROP.

While previous approaches include inference-time reconstruction search (e.g., ElasticTok) or externally specified reconstruction-quality targets (e.g., KARL), STROP learns to predict the token length from image features without an external target. The tokenizer uses one transformer to compress frozen DINOv3 features into an ordered sequence of discrete tokens and a second transformer to reconstruct spatial features from the retained tokens. The length predictor is trained through a four-stage curriculum using feature reconstruction errors and local rate distortion estimates.

The authors evaluate STROP on semantic segmentation, classification, and depth estimation. At a comparable average token budget, STROP outperforms separately trained fixed-length tokenizers on four segmentation benchmarks (VOC, COCO Stuff-27, ADE20K, Cityscapes). The paper also compares STROP with existing visual tokenizers and examines how token allocation relates to scene complexity. They show that predicted token lengths correlate with scene complexity, particularly on CLEVR and, to a lesser extent, COCO.

Critical strengths

Unlike several earlier visual tokenizers that primarily focus on image reconstruction or generation, STROP preserves semantic information through feature reconstruction. At comparable token budgets, the method outperforms separately trained fixed-length variants across four segmentation benchmarks (Table 1). It also achieves better segmentation results than FlexTok, One-D-Piece, and ALIT under a shared evaluation protocol (Table 2), although their training setups differ – please see the first weakness.

Furthermore , the authors provide a comprehensive evaluation of the proposed design, including fixed versus adaptive token lengths, causal ordering, reconstruction objectives, and curriculum strategies (Tables 1 and 3).

Critical weaknesses

Unfortunately, STROP and the fixed-length baselines differ in their training procedures (Table 1). In particular, STROP is trained with variable-length prefixes. This enables the model to place useful information in earlier tokens and may improve the learned representation independently of adaptive allocation. On the other hand, the baselines always use the same number of tokens. It therefore remains unclear how much of the improvement comes from adapting token lengths to individual images. Figure 9 provides useful additional comparisons, but does not fully isolate this effect. A comparison that randomly reassigns predicted lengths across images while preserving the overall token budget would help establish the benefit of adaptive allocation.

Furthermore, STROP’s difference from previous length prediction methods is its use of feature reconstruction errors and local rate-distortion estimates. However, the paper does not directly evaluate the contribution of local rate-distortion estimates over reconstruction errors alone.

Questions for authors

How much of STROP’s improvement comes from adapting token lengths to individual images vs variable-length prefix training (see the first weakness). How does the model perform if you randomly reassign the predicted lengths across images while preserving the overall token budget?

Can you please explain what the specific contribution of the local rate-distortion estimates is compared to using feature reconstruction errors alone?

5 comments

Thank you for suggesting the length-permutation control. We ran it, and it sharpens our claim: image-specific length allocation improves segmentation with the representation fixed, but it explains only part of the gain over fixed length. We also ran the error-only comparison you asked about, which shows what the probes contribute: they set the operating point. Details follow below.

Image-specific allocation

How does the model perform if you randomly reassign the predicted lengths across images while preserving the overall token budget?

We ran this as an inference-time test. We froze the adaptive tokenizer and its fitted segmentation readout, so the representation, including its exposure to variable prefixes, stays fixed. We consider two controls, each using exactly the same token count as the reference setup:

  • Shuffled: each crop keeps its own program but receives the predicted length of another crop, under a random permutation over all crops in the validation set (across all images in that set). The length distribution remains therefore intact.
  • Near-uniform: every crop receives ⌊Lˉ⌋\lfloor \bar{L} \rfloor tokens, where Lˉ\bar{L} is the mean predicted length, and a randomly chosen set of crops, just large enough to reach the exact total, receives one more.

The table below presents the impact of these randomizations in terms of change of the program-readout mIoU [percentage points] with respect to the STROP (mean ± sample s.d. over three tokenizer seeds; each control averages ten assignments).

DatasetPredicted (STROP) − shuffledPredicted (STROP) − near-uniform
VOC+2.02 ± 0.20+0.94 ± 0.25
COCO-Stuff27+1.11 ± 0.05+0.28 ± 0.02
ADE20K+0.51 ± 0.17+0.10 ± 0.14
Cityscapes+0.12 ± 0.12−0.06 ± 0.03

Lengths are assigned per sliding-window crop and crop predictions are combined before scoring, so a permutation changes both an image's total number of tokens and its split among crops.

On VOC and COCO-Stuff27, giving each crop its own length helps (per-seed 95% confidence intervals from a paired bootstrap over whole images exclude zero), and the gain over near-uniform holds in all three seeds. On ADE20K the gain over near-uniform is within seed variation, and on Cityscapes, near-uniform is slightly better.

These gains over near-uniform are smaller than the 1.6–2.1-point difference between adaptive STROP and fixed K=25 in Table 1, so part of that difference comes from how the tokenizer was trained, most likely from the variable-length prefixes. We agree that the sentence in Section 4 attributing the whole gain to allocation is too strong, and we will revise it to name both contributing factors (space permitting, with the results above). Both belong to STROP: the prefix curriculum teaches the generator to put the most useful content first; the length head decides, per crop, how much of it to keep.

(Continued from the previous comment.)

Length supervision and the training budget

The length target has two parts:

  • A per-crop part: the crop's teacher-feature reconstruction error divided by the running mean error. Crops that reconstruct worse than average get longer targets.
  • A probe factor m (Eqs. 3-5), shared by all crops, from the local probes. At a crop's current length L, we measure how much the error changes at L−δL-\delta and at L+δL+\delta; from running averages of these relative slopes, m falls below 1 when the error curve is flat around L and rises above 1 when extra tokens still reduce the error.

In the manuscript the base target is βK\beta K times the relative error, with β=0.75\beta = 0.75 (we have now noticed that the same symbol denotes the VQ commitment weight in Section 4 – our error; we will rename one of them).

The motivation of this formulation is that error alone is sufficient to decide which images need longer or shorter programs than others, but not how long the programs should be overall. Setting this overall level is the role of m (Eqs. 3-5). In other words, error alone fixes the relative lengths, but not their scale: the mean target is then βK\beta K, set by hand.

To corroborate the validity of the above rationale, we ran a new comparison: three 400k-update continuations from a common checkpoint (single tokenizer seed). The reference run is our default model, in which the probe factor m is computed during training; the factor settled to about 0.5, giving a mean training length of about 24 tokens.

  • Without the probe factor (m=1), the error target settled at about 48 training tokens, twice the reference's 24, with the lower reconstruction loss – which is expected at twice the rate.
  • With the factor fixed at 0.5, the value at which the probe factor settled in the reference run under our default settings, the model reproduced the reference length and validation loss.

So the probe factor is what places the model at its operating point. Once that value is known, a fixed scale is a simpler variant of the same length-head training. In both above variants, the head predicts an image-dependent length in a single pass, with no quality target at inference, and the achieved rate follows the data: the ImageNet-trained model, rate-matched at about 250 bits on ImageNet validation crops, spends 269–280 bits on COCO (Table 9).

We will describe the relation between the two variants in Section 2, and rewrite that section so the curriculum, the length target and inference are described separately, with the role of the probe factor m made more explicit.

Thank you for running both experiments and for the candid answers. I keep my rating for now, because the results confirm both concerns.

Image-specific allocation: in Table 1 (program rows), STROP beats fixed K=25 by 1.6-2.1 mIoU. In your new experiment, per-image lengths perform better than near-uniform lengths by 0.94 mIoU on VOC and 0.28 on COCO-Stuff-27. On ADE20K and Cityscapes, there is no measurable gain. So per-image lengths explain about half of the Table 1 gain on VOC, and little elsewhere. The rest likely comes from prefix training.

Length supervision: the probe factor is one value shared by all crops, and fixing m=0.5 reproduces the model (same length and validation loss). So the probes set the overall scale and give no per-image signal. The per-image signal is the relative reconstruction error alone. Appendix A (lines 728-730) notes that this error mixes image difficulty with length adequacy.

Minor points

  • In references, the entry Harari et al. (2025) is a placeholder (line 540). I believe the intended paper is Gizdov, Ullman, and Harari (CVPR 2025).
  • BSQ is cited to the Spherical Leech Quantization paper. The BSQ paper is Zhao, Xiong, and Krähenbühl (ICLR 2025): https://openreview.net/pdf?id=yGnsH3gQ6U
  • Table 14 repeats the COCO numbers from Table 13 under an ImageNet caption.

P.S. I know little time is left in the rebuttal period. I will read your reply before I enter my final rating during the discussion period.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the discussion and revisions

We thank the reviewers for a detailed discussion. The controls the reviewers proposed let us attribute STROP's gains more precisely; below is what the revised paper will state, with the evidence in our replies to each reviewer.

In short, on VOC and COCO-Stuff27 STROP’s gain over fixed K=25 does not depend on training lengths matched to content, and training on variable-length prefixes is a plausible explanation; the length head contributes a rate chosen per image in one pass, lower reconstruction error at the same token budget, and a length that itself describes the scene – benefits that segmentation shows least.

1. Table 1, STROP against fixed K=25 (DXCD, ijYv). On VOC and COCO-Stuff27, the shuffled-length continuation, trained from the common checkpoint with predicted lengths permuted across the batch in its last phase, reaches the same segmentation scores at 24 tokens per crop as the reference run at its own predicted lengths. On these two datasets STROP’s gain therefore does not depend on training with lengths matched to content; training on variable-length prefixes is a plausible explanation, although this control cannot measure its share. Permuting the lengths also makes the model insensitive to how lengths are assigned to crops, and the revision will include it as a variant of STROP.

2. What the length head contributes (DXCD).

  • Rate. It chooses each image's length in one pass, with no quality target at inference, and the rate follows the data: the ImageNet-trained model, rate-matched at about 250 bits on ImageNet, spends 269–280 bits on COCO (Table 9).
  • Allocation. At the same token budget, its choices lower teacher-feature reconstruction error by about 4% relative to near-uniform lengths on VOC and COCO-Stuff27 (adaptive tokenizer), 72–73% of what an oracle with each crop's true error curve achieves. On these two datasets it also adds a small segmentation gain, which the revision reports with readouts refitted for each allocation. At this token budget, allocation has little room to act: even the oracle lowers reconstruction error by only 5.5% on VOC and 1.5% on Cityscapes. A task tied to scene complexity shows a clearer effect: on CLEVR, with object-count readouts trained for each allocation, the length head lowers object-count error from 0.82 (near-uniform) to 0.74.
  • Description. On CLEVR, length correlates with object count (r0.74), and the length alone lets a frozen LLM lower object-count error from 2.01 (naive baseline) to 1.44.

3. Length supervision (DXCD). The probe factor m sets only the overall rate; the per-crop signal is relative reconstruction error. The local probes place the model at its operating point without a hand-set mean length (in single-seed continuations without them, training settles at about 48 tokens instead of 24), and the factor fixed at 0.5 reproduces the reference run once that value is known. Section 2 and Appendix A will be updated accordingly.

4. Table 2, STROP against public tokenizers (ijYv, dbad). Allowing pixel-reconstruction gradients into the STROP generator costs 7.9 latent readout and 8.6 program readout mIoU points (Table 3), which makes the training objective a plausible explanation for the gap. Table 2 measures segmentation information per bit; we do not attribute its gap to adaptive length.

5. Language models (ijYv). The length-only control matches the frozen LLM's zero-shot CLEVR result within paired 95% intervals, so Appendix M will no longer credit the codes with it. Trained linear readouts show that the codes carry object-count information beyond the length (0.74 vs. 1.29 from length alone; Table 20).

6. KARL, compute and terminology (dbad). The revision will include the distinction from KARL in supervision and inference, the compute and latency of prefix generation, and a definition of "program" with the limits of the semantic evidence.

7. Errors found by DXCD. The revision will correct the two citations and Table 14.

8. Next steps. The length head’s benefits appear most clearly where length tracks scene complexity, as in CLEVR object counting, and we will extend that evaluation beyond segmentation. Because the length head rarely assigns long programs, long prefixes are seldom trained; training over a wider range of lengths may strengthen the tokens they add.

We are again grateful for the controls the reviewers proposed. They made every claim above more precise.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.