acceptodds
Under review as a conference paper at ICLR 2027

Dense Segmentation as First-Order Output of Vision-Language Models

Abstract

Autoregressive vision-language models (VLMs) exhibit fine-grained spatial reasoning when describing images in words. However, producing the corresponding segmentation masks usually requires external segmentation models or learned mask decoders. This work introduces FOMO (first-order mask output) a geometry language that makes dense segmentation a native autoregressive output. FOMO's deterministic encoder uses adaptive quadtrees and pre-defined geometry stencils to describe binary masks as token sequences. Homogeneous regions are compactly encoded with only a few tokens, while additional tokens are allocated to complex boundary regions. FOMO sequences generated by VLMs deterministically form a mask without neural decoders. Theoretical proofs and empirical experiments demonstrate the codec's near-exact reconstruction, a Spearman correlation of between token budget and mask quality, and graceful degradation under most grammar-preserving token corruptions. Moreover, with the FOMO vocabulary extension and fine-tuning via low-rank adaptation, Qwen3.5-9B learns to generate masks for referring-expression segmentation queries without requiring any changes to its vision encoder or added external modules. Our method achieves 76.0%, 71.7%, and 71.7% cumulative IoU for RefCOCO val, RefCOCO+ val, and RefCOCOg val, using fixed geometric reconstruction without an external segmentation model or learned mask decoder.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
segmentation, vision-language models, reasoning segmentation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_G8FMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces FOMO, a geometry language that lets vision-language models generate segmentation masks as token sequences using adaptive quadtrees and fixed geometric stencils. These sequences are decoded without a neural mask decoder. The paper evaluates reconstruction quality, robustness to token corruption, and referring-expression segmentation using an adapted Qwen3.5-9B.

Critical strengths
  • Compact and interpretable representation. Adaptive quadtrees and geometric stencils achieve 0.991 reconstruction mIoU with an average of 222 tokens, demonstrating accurate and compact mask encoding.
  • Practical feasibility. Adapting Qwen3.5-9B with 129M trainable parameters produces meaningful results across three referring-expression segmentation benchmarks without a neural mask decoder.
Critical weaknesses
  • Insufficient controlled comparisons. Table 1 uses different backbones and training protocols. Without matched comparisons against polygon, RLE, or uniform-grid representations and component ablations, FOMO’s advantages over simpler alternatives remain unclear.

  • Limited robustness evidence. The experiments use selected grammar-preserving corruptions, including best-fitting stencil replacements, which may not reflect actual model errors. The theoretical guarantees also do not fully cover the deployed decoding procedure.

Questions for authors
  • Do you have comparisons against alternative representations using the same backbone, training data, and adaptation budget?

  • Does split suppression select replacement stencils using ground truth, and how closely do the tested corruptions reflect actual generation errors?

  • What are the actual generated sequence lengths and end-to-end inference latency, including mask reconstruction?

7 comments

We thank the reviewer for their thorough review and helpful comments. All three reviewers shared a major concern about whether our proposed FOMO method is able to outperform previously proposed neural-decoder-free baselines when using a similar model backbone and training regimen. Moreover, the reviewers requested ablations of our components, measurements of sequence length and latency, and an evaluation of robustness to the model’s own errors.

In response, we made a substantial effort to address each point. We apologize for the relatively late response, as we had to run several new experiments. However, we are confident that we were able to address essentially all identified issues and would welcome a re-evaluation of our work in light of the new findings presented.

Do you have comparisons against alternative representations using the same backbone, training data, and adaptation budget?

In response to the main shared reviewer concern, we have reimplemented the two most relevant neural-decoder-free baselines, PolyFormer and Text4Seg++, which encode segmentation masks as polygon vertices and via run-length encoding, respectively. Since our base model Qwen3.5-9B generates discrete tokens and does not support PolyFormer's regression decoder for continuous coordinates, our reimplementation writes one token per integer coordinate. Subsequently, we compared them to FOMO using the same Qwen3.5-9B backbone, the same training images, and the same LoRA and schedule. As shown in the table below, we find that under matched training conditions, FOMO outperforms both methods on all eight splits. For Text4Seg++, which predicts the bounding box first and uses an upsampled input (see below), the matched comparison is FOMO with the same two design choices (last two rows).

RefCOCO valtestAtestBRefCOCO+ valtestAtestBRefCOCOg valtestMean
PolyFormer67.6470.7563.5263.7967.9657.6163.5363.7164.81
Text4Seg++ (bounding box first, upsampled input)74.9177.5272.7070.4075.3166.6171.4771.3972.54
FOMO, as submitted76.0578.8372.9871.7476.1666.5571.7372.6873.34
FOMO, bounding box first77.6879.4075.2273.4677.3869.1273.6374.5675.06
FOMO, bounding box first, upsampled input79.4281.4276.6775.1179.3169.7875.3175.5476.57
FOMO, bounding box first, upsampled input, δ=4\delta=479.3181.4677.2274.9978.8570.6876.0976.2276.85

The LoRA adapters are identical in all runs (86.6M parameters).

The reimplemented version of Text4Seg++ scores noticeably lower than its published results. Our reimplementation achieves a mean cIoU of 72.5 across all eight splits, compared to the reported 79.3. Naturally, we cannot rule out mistakes in our reimplementation, as the Text4Seg++ code is not publicly available. However, we believe that the discrepancy is caused by dataset leakage. Text4Seg++ trains on the full COCO panoptic train split, and the val and test images of RefCOCO, RefCOCO+ and RefCOCOg are taken from COCO train. Consequently, the Text4Seg++ model has likely seen the evaluation masks during training, although not the referring expressions. Conversely, FOMO and the reimplemented baselines have not been trained on the COCO panoptic dataset, and we removed RefCOCO-family training samples that overlap with the val and test splits.

While reimplementing Text4Seg++, we noticed that two of its design choices are equally compatible with our method. First, it predicts the bounding box before the mask. Second, the input image is upsampled to provide the encoder with more fine-grained image features. Both of these design choices can easily be integrated into FOMO and were found to improve its performance. Predicting a bounding box first gains +1.7 cIoU, and upsampling the input yields another +1.5 (mean over the eight splits: 73.34 → 75.06 → 76.57).

At the same time, the reviewers correctly noted that performance gains should not be measured by segmentation performance alone but must be assessed together with sequence length, which directly correlates with computational cost and latency. While running these experiments, we noticed that relaxing FOMO's stencil matching tolerance δ\delta greatly shortens sequence length, with negligible impact on segmentation accuracy in trained vision-language models.

The table below shows that the mean sequence lengths of PolyFormer, Text4Seg++, and FOMO with the bounding box first and δ=4\delta=4 are highly similar. Notably, VLMs are not able to output segmentation masks encoded as polygon vertices as efficiently as the theoretical advantages of this format would suggest. All cIoU values are means over the eight splits.

cIoU (encoded-decoded)Mean tokens (encoded)Trained model cIoUMean tokens generated
PolyFormer99.367664.81141
Text4Seg++95.1414172.54139
FOMO, as submitted, δ=1.5\delta=1.599.2722473.34279
FOMO, bounding box first, δ=1.5\delta=1.599.27251 (27 of them box text)75.06297
FOMO, bounding box first, δ=4\delta=497.53126 (27 of them box text)74.85136
FOMO, bounding box first, upsampled input, δ=4\delta=497.53126 (27 of them box text)76.85132

The reviewer also asked for component ablations and a uniform-grid representation. We found that a model trained without stencils (signature-only quadtree) reaches the same accuracy (73.4 against 73.3 cIoU over the eight splits) but requires about 22% more tokens and 26% more time, so the stencils significantly shorten the sequence without impacting representation quality.

A uniform subdivision of our tree needs 2,167 tokens for the reconstruction quality that the adaptive tree reaches with 318, which exceeds the 2,048-token budget, so it cannot be trained under the matched recipe. We view adaptivity as a core feature of FOMO that is highly beneficial for its application to real-world images. Alternatively, one can reduce the resolution of the image grid to facilitate uniform subdivision at a reasonable sequence length. A 64$\times$64 grid corresponds to Text4Seg++, which is shown to perform worse than FOMO by 4.0 cIoU under matched conditions (72.54 against 76.57 over the eight splits).

The ablation of the Laplacian completion is discussed at the end of this response.

Reviewer_isDpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FOMO represents binary masks using adaptive quadtrees and a frozen, data-derived bank of 4,096 TSDF stencils, with fixed geometric reconstruction and Laplacian completion. The paper provides conditional codec guarantees and adapts Qwen3.5-9B through vocabulary expansion and LoRA (129M trainable parameters). On 1,000 RefCOCO+ validation masks, the codec achieves 0.991 mean reconstruction IoU at 222 tokens on average; generated masks reach 76.0/71.7/71.7 validation cIoU on RefCOCO/+/g.

Critical strengths
  • The adaptive geometry representation achieves high reconstruction fidelity with fixed reconstruction rules. The reported 0.991 mean IoU at 222 tokens provides concrete evidence that a neural mask decoder is unnecessary for faithful mask representation.
  • The evaluation distinguishes codec fidelity from generated-mask accuracy, and the appendix explicitly excludes evaluation images across overlapping training sources. Together with clearly qualified theoretical guarantees, this strengthens the credibility of the feasibility results.
Critical weaknesses
  • The incremental contribution over existing native mask generation is not established: SimpleSeg and Text4Seg already provide direct outputs without neural mask decoders. Text4Seg++ is cited but omitted from Table 1; its unrefined Qwen2-7B configuration reports 81.5/76.9/79.6 validation cIoU (Table II). Protocol differences preclude causal conclusions, but this omission leaves the central comparison incomplete.
  • Experiments do not isolate the proposed geometry language’s benefits: There are no comparisons using the same backbone/data against RLE or polygon representations, or ablations of stencils, adaptive subdivision, and completion. Codec budget sweeps do not measure generation learnability; corruption tests use ground-truth streams, including best-stencil replacement for suppressed splits. Without measured generation latency and reconstruction cost, neither a superior accuracy–cost trade-off nor robustness under actual generation errors is demonstrated.
Questions for authors
  • Decisive additional evidence: Under the same Qwen3.5-9B adaptation, training data, and image resolution, how does FOMO compare with Text4Seg++-style RLE/bricks and polygon outputs in cIoU, generated sequence length, and end-to-end latency?
  • Decisive ablations: What performance remains with a signature-only quadtree, uniform subdivision, or no Laplacian completion? These would identify which components justify the added complexity.
  • Decisive robustness evidence: Does robustness to split/stencil errors persist with model-sampled replacements instead of target-mask-matched alternatives, and how sensitive are generated masks to tile/sign mistakes?
7 comments

We thank the reviewer for their thorough review and helpful comments. All three reviewers shared a major concern about whether our proposed FOMO method is able to outperform previously proposed neural-decoder-free baselines when using a similar model backbone and training regimen. Moreover, the reviewers requested ablations of our components, measurements of sequence length and latency, and an evaluation of robustness to the model’s own errors.

In response, we made a substantial effort to address each point. We apologize for the relatively late response, as we had to run several new experiments. However, we are confident that we were able to address essentially all identified issues and would welcome a re-evaluation of our work in light of the new findings presented.

Question 1

Decisive additional evidence: Under the same Qwen3.5-9B adaptation, training data, and image resolution, how does FOMO compare with Text4Seg++-style RLE/bricks and polygon outputs in cIoU, generated sequence length, and end-to-end latency?

In response to the main shared reviewer concern, we have reimplemented the two most relevant neural-decoder-free baselines, PolyFormer and Text4Seg++, which encode segmentation masks as polygon vertices and via run-length encoding, respectively. Since our base model Qwen3.5-9B generates discrete tokens and does not support PolyFormer's regression decoder for continuous coordinates, our reimplementation writes one token per integer coordinate. Subsequently, we compared them to FOMO using the same Qwen3.5-9B backbone, the same training images, and the same LoRA and schedule. As shown in the table below, we find that under matched training conditions, FOMO outperforms both methods on all eight splits. For Text4Seg++, which predicts the bounding box first and uses an upsampled input (see below), the matched comparison is FOMO with the same two design choices (last two rows).

RefCOCO valtestAtestBRefCOCO+ valtestAtestBRefCOCOg valtestMean
PolyFormer67.6470.7563.5263.7967.9657.6163.5363.7164.81
Text4Seg++ (bounding box first, upsampled input)74.9177.5272.7070.4075.3166.6171.4771.3972.54
FOMO, as submitted76.0578.8372.9871.7476.1666.5571.7372.6873.34
FOMO, bounding box first77.6879.4075.2273.4677.3869.1273.6374.5675.06
FOMO, bounding box first, upsampled input79.4281.4276.6775.1179.3169.7875.3175.5476.57
FOMO, bounding box first, upsampled input, δ=4\delta=479.3181.4677.2274.9978.8570.6876.0976.2276.85

The LoRA adapters are identical in all runs (86.6M parameters).

The reimplemented version of Text4Seg++ scores noticeably lower than its published results. Our reimplementation achieves a mean cIoU of 72.5 across all eight splits, compared to the reported 79.3. Naturally, we cannot rule out mistakes in our reimplementation, as the Text4Seg++ code is not publicly available. However, we believe that the discrepancy is caused by dataset leakage. Text4Seg++ trains on the full COCO panoptic train split, and the val and test images of RefCOCO, RefCOCO+ and RefCOCOg are taken from COCO train. Consequently, the Text4Seg++ model has likely seen the evaluation masks during training, although not the referring expressions. Conversely, FOMO and the reimplemented baselines have not been trained on the COCO panoptic dataset, and we removed RefCOCO-family training samples that overlap with the val and test splits.

While reimplementing Text4Seg++, we noticed that two of its design choices are equally compatible with our method. First, it predicts the bounding box before the mask. Second, the input image is upsampled to provide the encoder with more fine-grained image features. Both of these design choices can easily be integrated into FOMO and were found to improve its performance. Predicting a bounding box first gains +1.7 cIoU, and upsampling the input yields another +1.5 (mean over the eight splits: 73.34 → 75.06 → 76.57).

At the same time, the reviewers correctly noted that performance gains should not be measured by segmentation performance alone but must be assessed together with sequence length, which directly correlates with computational cost and latency. While running these experiments, we noticed that relaxing FOMO's stencil matching tolerance δ\delta greatly shortens sequence length, with negligible impact on segmentation accuracy in trained vision-language models.

The table below shows that the mean sequence lengths of PolyFormer, Text4Seg++, and FOMO with the bounding box first and δ=4\delta=4 are highly similar. Notably, VLMs are not able to output segmentation masks encoded as polygon vertices as efficiently as the theoretical advantages of this format would suggest. All cIoU values are means over the eight splits.

cIoU (encoded-decoded)Mean tokens (encoded)Trained model cIoUMean tokens generated
PolyFormer99.367664.81141
Text4Seg++95.1414172.54139
FOMO, as submitted, δ=1.5\delta=1.599.2722473.34279
FOMO, bounding box first, δ=1.5\delta=1.599.27251 (27 of them box text)75.06297
FOMO, bounding box first, δ=4\delta=497.53126 (27 of them box text)74.85136
FOMO, bounding box first, upsampled input, δ=4\delta=497.53126 (27 of them box text)76.85132

The table also confirms the reviewer's point that codec budget sweeps do not measure generation learnability: polygon vertices are the best code and the worst trained model, while FOMO with δ=4\delta=4 is a less exact code than FOMO with δ=1.5\delta=1.5 and trains equally well.

We find that end-to-end inference latency on an NVIDIA A100 GPU is highly correlated with sequence length. Mask reconstruction only requires a negligible amount of time, owing to its lightweight deterministic design without the need for any neural network modules: autoregressive generation takes about 68 milliseconds per token for every format, and TSDF reconstruction about 80 milliseconds per mask, less than 1% of the total. In our experiments, Qwen3.5-9B with FOMO (bounding box first, upsampled input, δ=4\delta=4) requires an average of 10.1 seconds to predict and decode a mask, compared to 9.9 seconds for PolyFormer and 9.7 seconds for Text4Seg++. FOMO, as submitted, requires an average of 19.8 seconds, as its sequences are about twice as long.

Reviewer_6YR7OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how VLMs can directly produce pixel-level segmentation results as autoregressive token sequences, without external segmentation models such as SAM or neural mask decoders. To this end, the authors propose FOMO (first-order mask output), a mask representation language with fixed geometric meanings.

Specifically, the encoder first converts a binary mask into a truncated signed distance field (TSDF), constructs adaptive quadtrees within tiles near the boundary, and represents the mask using region signs, child-node structures, and local geometric templates (stencils). The stencil bank is constructed by clustering training masks and applying geometric transformations, and is subsequently frozen. The model generates these tokens in a coarse-to-fine order, while a deterministic decoder reconstructs the mask through fixed rules and Laplacian inpainting, without a neural mask decoder.

The authors separately evaluate the codec’s reconstruction fidelity and the VLM’s segmentation prediction performance. On 1,000 masks from the RefCOCO+ validation set, the selected configuration uses an average of 222 tokens per mask and achieves a mean reconstruction IoU of 0.991. After adding the geometry vocabulary to Qwen3.5-9B and training LoRA adapters, new token embeddings, and a geometry output head on 810,855 segmentation instructions, the model achieves cumulative IoU (cIoU) scores of 76.0, 71.7, and 71.7 on the RefCOCO, RefCOCO+, and RefCOCOg validation sets, respectively.

Critical strengths
  1. The paper proposes an interpretable geometric representation that can be learned directly through next-token prediction, offering clear methodological value. Adaptive quadtrees allocate more representational capacity to complex boundaries, while stencil tokens correspond to explicit local TSDFs rather than implicit indices whose meanings depend on a neural decoder.

  2. The authors demonstrate both the ability to encode ground-truth masks and the feasibility of teaching a VLM this representation language. With pretrained weights frozen, training only 129M parameters achieves the segmentation results reported in Table 1, demonstrating that existing VLMs can learn this explicit grammar.

Critical weaknesses
  1. The paper does not sufficiently establish the advantages of its proposed representation over existing native mask-output methods. Section 2 discusses approaches such as polygons, Text4Seg, and Text4Seg++ that can directly represent masks. Therefore, eliminating a neural mask decoder alone is insufficient to establish the main contribution; the key question is whether the adaptive geometry language is more compact, easier to learn, or more robust. The paper lacks controlled comparisons using the same backbone and training data, as well as end-to-end latency measurements covering both autoregressive generation and deterministic reconstruction. This makes it difficult to assess the practical benefits of the representation over simpler alternatives.

  2. A clear segmentation performance gap remains relative to state-of-the-art methods, leaving its practical competitiveness uncertain. In Table 1, the proposed method still trails leading approaches on referring expression segmentation. Although it requires neither an external segmentation model nor a learnable mask decoder, without inference efficiency comparisons, it remains unclear whether this architectural simplification compensates for the loss in segmentation accuracy.

Questions for authors
  1. Regarding the added value of the representation: Could the authors compare FOMO with at least one direct mask representation, such as RLE or polygons, using the same Qwen3.5-9B backbone, training data, and trainable parameter budget? Please also report token counts at matched ground-truth mask reconstruction IoU and results using a pure quadtree representation without the stencil bank. These comparisons would directly inform the assessment of the adaptive geometry language’s contribution.

  2. Regarding practical efficiency: Please report the median and upper percentiles of actual generated sequence lengths, the proportion of outputs affected by the 2,048-token budget constraint, and the time spent on autoregressive generation and TSDF reconstruction under identical hardware and input settings.

  3. Does segmentation training affect the model’s original multimodal understanding and language capabilities? Although the original backbone weights are frozen, LoRA still changes the model’s behavior. Comparisons before and after training on general multimodal understanding and text-only tasks would help determine whether acquiring segmentation capabilities affects its existing abilities.

6 comments

We thank the reviewer for their thorough review and helpful comments. All three reviewers shared a major concern about whether our proposed FOMO method is able to outperform previously proposed neural-decoder-free baselines when using a similar model backbone and training regimen. Moreover, the reviewers requested ablations of our components, measurements of sequence length and latency, and an evaluation of robustness to the model’s own errors.

In response, we made a substantial effort to address each point. We apologize for the relatively late response, as we had to run several new experiments. However, we are confident that we were able to address essentially all identified issues and would welcome a re-evaluation of our work in light of the new findings presented.

Question 1

Regarding the added value of the representation: Could the authors compare FOMO with at least one direct mask representation, such as RLE or polygons, using the same Qwen3.5-9B backbone, training data, and trainable parameter budget? Please also report token counts at matched ground-truth mask reconstruction IoU and results using a pure quadtree representation without the stencil bank. These comparisons would directly inform the assessment of the adaptive geometry language's contribution.

In response to the main shared reviewer concern, we have reimplemented the two most relevant neural-decoder-free baselines, PolyFormer and Text4Seg++, which encode segmentation masks as polygon vertices and via run-length encoding, respectively. Since our base model Qwen3.5-9B generates discrete tokens and does not support PolyFormer's regression decoder for continuous coordinates, our reimplementation writes one token per integer coordinate. Subsequently, we compared them to FOMO using the same Qwen3.5-9B backbone, the same training images, and the same LoRA and schedule. As shown in the table below, we find that under matched training conditions, FOMO outperforms both methods on all eight splits. For Text4Seg++, which predicts the bounding box first and uses an upsampled input (see below), the matched comparison is FOMO with the same two design choices (last two rows).

RefCOCO valtestAtestBRefCOCO+ valtestAtestBRefCOCOg valtestMean
PolyFormer67.6470.7563.5263.7967.9657.6163.5363.7164.81
Text4Seg++ (bounding box first, 64x64)73.4676.4171.4069.4073.5665.1769.8170.4271.20
Text4Seg++ (bounding box first, 64x64, upsampled input)74.9177.5272.7070.4075.3166.6171.4771.3972.54
FOMO, as submitted76.0578.8372.9871.7476.1666.5571.7372.6873.34
FOMO, bounding box first77.6879.4075.2273.4677.3869.1273.6374.5675.06
FOMO, bounding box first, upsampled input79.4281.4276.6775.1179.3169.7875.3175.5476.57
FOMO, bounding box first, upsampled input, δ=479.3181.4677.2274.9978.8570.6876.0976.2276.85

The LoRA adapters are identical in all runs (86.6M parameters).

The reimplemented version of Text4Seg++ scores noticeably lower than its published results. Our reimplementation achieves a mean cIoU of 72.5 across all eight splits, compared to the reported 79.3. Naturally, we cannot rule out mistakes in our reimplementation, as the Text4Seg++ code is not publicly available. However, we believe that the discrepancy is caused by dataset leakage. Text4Seg++ trains on the full COCO panoptic train split, and the val and test images of RefCOCO, RefCOCO+ and RefCOCOg are taken from COCO train. Consequently, the Text4Seg++ model has likely seen the evaluation masks during training, although not the referring expressions. Conversely, FOMO and the reimplemented baselines have not been trained on the COCO panoptic dataset, and we removed RefCOCO-family training samples that overlap with the val and test splits.

While reimplementing Text4Seg++, we noticed that two of its design choices are equally compatible with our method. First, it predicts the bounding box before the mask. Second, the input image is upsampled to provide the encoder with more fine-grained image features. Both of these design choices can easily be integrated into FOMO and were found to improve its performance. Predicting a bounding box first gains +1.7 cIoU, and upsampling the input yields another +1.5 (mean over the eight splits: 73.34 → 75.06 → 76.57).

At the same time, the reviewers correctly noted that performance gains should not be measured by segmentation performance alone but must be assessed together with sequence length, which directly correlates with computational cost and latency. While running these experiments, we noticed that relaxing FOMO's stencil matching tolerance δ\delta greatly shortens sequence length, with negligible impact on segmentation accuracy in trained vision-language models.

The table below shows that the mean sequence lengths of PolyFormer, Text4Seg++, and FOMO with the bounding box first and δ=4\delta=4 are highly similar. Notably, VLMs are not able to output segmentation masks encoded as polygon vertices as efficiently as the theoretical advantages of this format would suggest. All cIoU values are means over the eight splits.

cIoU (encoded-decoded)Mean tokens (encoded)Trained model cIoUMean tokens generated
PolyFormer99.367664.81141
Text4Seg++95.1414172.54139
FOMO, as submitted, δ=1.5\delta=1.599.2722473.34279
FOMO, bounding box first, δ=1.5\delta=1.599.27251 (27 of them box text)75.06297
FOMO, bounding box first, δ=4\delta=497.53126 (27 of them box text)74.85136
FOMO, bounding box first, upsampled input, δ=4\delta=497.53126 (27 of them box text)76.85132

Finally, we trained a FOMO version without the stencil bank, whose quadtree uses only inside/outside signatures. FOMO with this signature-only quadtree reaches a similar accuracy (+0.2 ± 0.3 cIoU over eight splits), but requires 22% more tokens and 26% more time during inference.

At a matched mean reconstruction IoU of about 0.965 on the 1,000 RefCOCO+ val masks used in the paper, FOMO with δ=4\delta=4 needs 99 tokens per mask, a Text4Seg++ grid of 128$\times$128 cells needs 298, the quadtree without stencils needs 318, and a uniform subdivision needs 2,167. Polygon vertices need 75 tokens at 0.992, but the trained model does not reach this quality (see the table above).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.