- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces FOMO, a geometry language that lets vision-language models generate segmentation masks as token sequences using adaptive quadtrees and fixed geometric stencils. These sequences are decoded without a neural mask decoder. The paper evaluates reconstruction quality, robustness to token corruption, and referring-expression segmentation using an adapted Qwen3.5-9B.
Critical strengths
- Compact and interpretable representation. Adaptive quadtrees and geometric stencils achieve 0.991 reconstruction mIoU with an average of 222 tokens, demonstrating accurate and compact mask encoding.
- Practical feasibility. Adapting Qwen3.5-9B with 129M trainable parameters produces meaningful results across three referring-expression segmentation benchmarks without a neural mask decoder.
Critical weaknesses
-
Insufficient controlled comparisons. Table 1 uses different backbones and training protocols. Without matched comparisons against polygon, RLE, or uniform-grid representations and component ablations, FOMO’s advantages over simpler alternatives remain unclear.
-
Limited robustness evidence. The experiments use selected grammar-preserving corruptions, including best-fitting stencil replacements, which may not reflect actual model errors. The theoretical guarantees also do not fully cover the deployed decoding procedure.
Questions for authors
-
Do you have comparisons against alternative representations using the same backbone, training data, and adaptation budget?
-
Does split suppression select replacement stencils using ground truth, and how closely do the tested corruptions reflect actual generation errors?
-
What are the actual generated sequence lengths and end-to-end inference latency, including mask reconstruction?
7 comments
Do you have comparisons against alternative representations using the same backbone, training data, and adaptation budget?
In response to the main shared reviewer concern, we have reimplemented the two most relevant neural-decoder-free baselines, PolyFormer and Text4Seg++, which encode segmentation masks as polygon vertices and via run-length encoding, respectively. Since our base model Qwen3.5-9B generates discrete tokens and does not support PolyFormer's regression decoder for continuous coordinates, our reimplementation writes one token per integer coordinate. Subsequently, we compared them to FOMO using the same Qwen3.5-9B backbone, the same training images, and the same LoRA and schedule. As shown in the table below, we find that under matched training conditions, FOMO outperforms both methods on all eight splits. For Text4Seg++, which predicts the bounding box first and uses an upsampled input (see below), the matched comparison is FOMO with the same two design choices (last two rows).
| RefCOCO val | testA | testB | RefCOCO+ val | testA | testB | RefCOCOg val | test | Mean | |
|---|---|---|---|---|---|---|---|---|---|
| PolyFormer | 67.64 | 70.75 | 63.52 | 63.79 | 67.96 | 57.61 | 63.53 | 63.71 | 64.81 |
| Text4Seg++ (bounding box first, upsampled input) | 74.91 | 77.52 | 72.70 | 70.40 | 75.31 | 66.61 | 71.47 | 71.39 | 72.54 |
| FOMO, as submitted | 76.05 | 78.83 | 72.98 | 71.74 | 76.16 | 66.55 | 71.73 | 72.68 | 73.34 |
| FOMO, bounding box first | 77.68 | 79.40 | 75.22 | 73.46 | 77.38 | 69.12 | 73.63 | 74.56 | 75.06 |
| FOMO, bounding box first, upsampled input | 79.42 | 81.42 | 76.67 | 75.11 | 79.31 | 69.78 | 75.31 | 75.54 | 76.57 |
| FOMO, bounding box first, upsampled input, | 79.31 | 81.46 | 77.22 | 74.99 | 78.85 | 70.68 | 76.09 | 76.22 | 76.85 |
The LoRA adapters are identical in all runs (86.6M parameters).
The reimplemented version of Text4Seg++ scores noticeably lower than its published results. Our reimplementation achieves a mean cIoU of 72.5 across all eight splits, compared to the reported 79.3. Naturally, we cannot rule out mistakes in our reimplementation, as the Text4Seg++ code is not publicly available. However, we believe that the discrepancy is caused by dataset leakage. Text4Seg++ trains on the full COCO panoptic train split, and the val and test images of RefCOCO, RefCOCO+ and RefCOCOg are taken from COCO train. Consequently, the Text4Seg++ model has likely seen the evaluation masks during training, although not the referring expressions. Conversely, FOMO and the reimplemented baselines have not been trained on the COCO panoptic dataset, and we removed RefCOCO-family training samples that overlap with the val and test splits.
While reimplementing Text4Seg++, we noticed that two of its design choices are equally compatible with our method. First, it predicts the bounding box before the mask. Second, the input image is upsampled to provide the encoder with more fine-grained image features. Both of these design choices can easily be integrated into FOMO and were found to improve its performance. Predicting a bounding box first gains +1.7 cIoU, and upsampling the input yields another +1.5 (mean over the eight splits: 73.34 → 75.06 → 76.57).
At the same time, the reviewers correctly noted that performance gains should not be measured by segmentation performance alone but must be assessed together with sequence length, which directly correlates with computational cost and latency. While running these experiments, we noticed that relaxing FOMO's stencil matching tolerance greatly shortens sequence length, with negligible impact on segmentation accuracy in trained vision-language models.
The table below shows that the mean sequence lengths of PolyFormer, Text4Seg++, and FOMO with the bounding box first and are highly similar. Notably, VLMs are not able to output segmentation masks encoded as polygon vertices as efficiently as the theoretical advantages of this format would suggest. All cIoU values are means over the eight splits.
| cIoU (encoded-decoded) | Mean tokens (encoded) | Trained model cIoU | Mean tokens generated | |
|---|---|---|---|---|
| PolyFormer | 99.36 | 76 | 64.81 | 141 |
| Text4Seg++ | 95.14 | 141 | 72.54 | 139 |
| FOMO, as submitted, | 99.27 | 224 | 73.34 | 279 |
| FOMO, bounding box first, | 99.27 | 251 (27 of them box text) | 75.06 | 297 |
| FOMO, bounding box first, | 97.53 | 126 (27 of them box text) | 74.85 | 136 |
| FOMO, bounding box first, upsampled input, | 97.53 | 126 (27 of them box text) | 76.85 | 132 |
The reviewer also asked for component ablations and a uniform-grid representation. We found that a model trained without stencils (signature-only quadtree) reaches the same accuracy (73.4 against 73.3 cIoU over the eight splits) but requires about 22% more tokens and 26% more time, so the stencils significantly shorten the sequence without impacting representation quality.
A uniform subdivision of our tree needs 2,167 tokens for the reconstruction quality that the adaptive tree reaches with 318, which exceeds the 2,048-token budget, so it cannot be trained under the matched recipe. We view adaptivity as a core feature of FOMO that is highly beneficial for its application to real-world images. Alternatively, one can reduce the resolution of the image grid to facilitate uniform subdivision at a reasonable sequence length. A 64$\times$64 grid corresponds to Text4Seg++, which is shown to perform worse than FOMO by 4.0 cIoU under matched conditions (72.54 against 76.57 over the eight splits).
The ablation of the Laplacian completion is discussed at the end of this response.
We thank the reviewer for their thorough review and helpful comments. All three reviewers shared a major concern about whether our proposed FOMO method is able to outperform previously proposed neural-decoder-free baselines when using a similar model backbone and training regimen. Moreover, the reviewers requested ablations of our components, measurements of sequence length and latency, and an evaluation of robustness to the model’s own errors.
In response, we made a substantial effort to address each point. We apologize for the relatively late response, as we had to run several new experiments. However, we are confident that we were able to address essentially all identified issues and would welcome a re-evaluation of our work in light of the new findings presented.