acceptodds
Under review as a conference paper at ICLR 2027

Factorized Semantic IDs for Controllable Coarse-to-Fine Outfit Retrieval

Abstract

We study controllable outfit completion: retrieving a compatible item for a partial outfit while letting the caller specify what the returned item must be, its role (e.g. “shoes”) and, optionally, its coarse color, without scoring every catalog item at query time. Our approach gives every catalog item a short ID made of three independent fields: role and color, assigned by fixed rules, and a compatibility code that groups items a frozen, role-conditioned Transformer teacher treats as interchangeable for the same kinds of outfits. This compatibility code comes from clustering the teacher’s own item embeddings, not from a learned decoder. At query time, the caller’s role and color constraints are enforced first, by intersecting these fields the way a database query intersects indexes on several columns, before any embedding similarity is computed; only the resulting small candidate set is then reranked with the teacher’s continuous compatibility score. Under ground-truth color control, this factorized routing at L=8 retains 98.8% of a role+color oracle’s Recall@10 while scoring only 1,032 candidates (11.3×compression). Under counterfactual color control, that is, deliberately requesting a color different from the true target’s so the system must actually honor an unfamiliar constraint rather than default to what it would retrieve anyway, factorized routing at L=8 recovers 88.2% of the top-10 items the exhaustive constrained teacher would itself select, against 10.8% for a matched-budget random control; both results hold stably across 3 independently trained teacher seeds (Section 7). Unlike prior compatibility-ranking methods, which offer no native mechanism to enforce such constraints, and unlike autoregressive Semantic-ID methods, which require a decoder and a collaborative-filtering training signal that outfit-compatibility data does not provide, our factorized index adds this controllability to an existing compatibility scorer using only fixed rules and clustering, at a small fraction of the cost of scoring the full catalog.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Recommender Systems & Information Retrieval, Controllable & Constrained Retrieval, Discrete Representations & Quantization, Efficient Search & Candidate Generation, Contrastive Representation Learning, Fashion & Multimodal Recommendation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_fKdZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates controllable outfit completion—retrieving a missing compatible garment for a partial outfit subject to hard, caller-specified discrete constraints (such as garment role and coarse color) without scoring the entire catalog. To achieve this, the authors propose Factorized Semantic IDs (SIDs) combined with a coarse-to-fine routing and continuous reranking pipeline.

Critical strengths
  1. The paper presents a practical coarse-to-fine retrieval pipeline that applies categorical constraints before continuous scoring. It reduces the number of scored candidates by 11.3×11.3\times to 22.7×22.7\times while preserving 93.6%93.6\% to 98.8%98.8\% of the exhaustive oracle's Recall@10.
  2. The evaluation is well-constructed, using a counterfactual steerability protocol against a matched random control to verify constraint adherence, backed by query-clustered bootstrap confidence intervals and multiple teacher seeds.
  3. The authors provide useful per-role breakdowns, openly discuss seed-dependent limitations, and document negative results on learned codebook alternatives, which makes the empirical findings easier to trust and reproduce.
Critical weaknesses
  1. The paper frames its main contribution around "Factorized Semantic IDs" and cites generative retrieval work (e.g., TIGER, DSI). However, the actual method involves no generative modeling, autoregressive decoding, or discrete token prediction. stripped of the terminology, it assigns each item a metadata tag (category role rr), a pretrained discrete cluster label (opaque color code cc), and a kk-means centroid ID (kk). At inference time, it simply performs a Boolean inverted-index lookup (r∩cr \cap c), probes the nearest cluster centroids (similar to standard IVF search), and reranks with cosine similarity. Calling faceted search combined with vector quantization "Factorized Semantic ID routing" overstates the algorithmic novelty.
  2. In Eq. 2, the top-LL compatibility codes (TopL(z^(X);r)Top_L(\hat{z}(\mathcal{X}); r)) are selected over role rr's entire codebook without considering the target color cc, and the intersection with Cr,c\mathcal{C}_{r,c} only happens afterward. For rare role-color combinations, the top-L clusters often contain no items of the requested color, resulting in an empty candidate set (Cand_L(X, r, c) = empty). To fix this, the method relies on a fallback escalation ladder (L = 1 -> 2 -> 4 -> 8 -> C_{r,c}). Needing runtime escalation like this reflects a loose coupling between the routing metric and the hard constraints rather than a unified constrained-retrieval design.
  3. Throughout Sections 6 and 7, efficiency is measured only by candidate reduction ratios (∣CandL∣|Cand_L| versus the full partition size). In real retrieval systems, multi-index lookups, dynamic fallback branching, and non-contiguous memory gathers add non-trivial overhead. Without actual hardware-level latency (e.g., P95/P99 latency in milliseconds) and QPS numbers, it is hard to tell how much end-to-end speedup this pipeline achieves in practice.
  4. As described in Section 2, the color attribute is restricted to 7 opaque cluster codes from a pretrained model without human-interpretable semantics. In practical outfit completion, users usually search with fine-grained, open-vocabulary attributes (such as "burgundy", "oversized", or "linen"). Scaling this discrete partitioning approach to multiple fine-grained attributes would quickly run into combinatorial sparsity, causing frequent empty candidate sets and forcing the escalation policy to fall back to exhaustive scans.
Questions for authors

see weaknesses.

1 comment

As there was no further author discussion, I will maintain my original rating based on the submitted response and review.

Reviewer_F6FSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses controllable outfit completion: retrieving a compatible item for a partial outfit while letting the caller constrain the item's role (e.g. "shoes") and optionally its coarse color. The core idea is a factorized semantic ID—each catalog item gets a short, three-field ID: role (a fixed metadata lookup), color (read directly from a pretrained attribute encoder), and a compatibility code obtained by role-conditioned k-means over the item embeddings of a frozen, role-conditioned Transformer teacher. At query time, role/color constraints are enforced first, by intersecting these fields like a multi-column database index, before any similarity is computed; only the resulting small candidate set is reranked with the teacher's continuous score.

Critical strengths

1、easy to understand 2、Rigorous, insightful evaluation design.

Critical weaknesses

1、Limited methodological novelty. The pipeline is, at its core, a fixed role lookup + a read-off color code + a single k-means clustering of frozen teacher embeddings, followed by index intersection and reranking. There is no new learning objective and no new architecture; the teacher is explicitly not a contribution (FITB merely matches, not beats, OutfitTransformer). The work is best characterized as a systems/engineering integration over an existing scorer. 2、Missing the genuinely relevant baseline. The central claim—"enforce constraints before similarity scoring"—is exactly what filtered-ANN / attribute-filtered IVF (e.g. Faiss filtered search, ACORN) is designed for, and the paper itself concedes the ANN/IVF analogy. Yet no such baseline appears; comparisons are only against exhaustive oracle / random / code-only. Without a filtered-ANN comparison at matched budget, it is unclear how much of the "controllable retrieval" novelty is genuinely new versus a mature technique applied to a new task.

Questions for authors

Since the color field is a hard filter on a pretrained attribute code, what fraction of the observed controllability comes from this filter alone versus the learned compatibility code? An ablation isolating the color filter under counterfactual control would clarify the respective roles

Reviewer_QRdKOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper aims to enforce role and optional color constraints in outfit completion while reducing candidate-scoring costs.The paper assigns each item a Factorized Semantic ID containing role, color, and a compatibility code derived by clustering a frozen Transformer’s item embeddings. It selects the nearest compatibility clusters within the requested role, applies the optional color filter, and ranks the resulting candidate items by similarity to the query embedding.Experiments on Polyvore Outfits-ND evaluate retrieval quality and candidate-scoring reduction under ground-truth and counterfactual color constraints. The authors also compare k-means with two learned partitioning methods, finding that k-means performs better in the reported experimental setting.

Critical strengths
  1. The proposed method combines attribute constraints, compatibility clustering, and continuous reranking. This combination provides a practical approach to controllable outfit completion.
Critical weaknesses
  1. Insufficient motivation.
    The paper claims that existing methods lack effective mechanisms for enforcing hard constraints. However, this claim is neither adequately supported by the literature nor substantiated by targeted experiments demonstrating that existing methods indeed exhibit this limitation.

  2. Insufficiently Demonstrated Novelty.
    The proposed factorized index appears conceptually similar to combining metadata filtering with standard ANN retrieval. Although the paper briefly discusses this distinction, it lacks direct comparisons with relevant baselines, such as separate ANN indexes for role/color partitions using the same teacher embeddings. Without these comparisons, it is difficult to assess the incremental contribution and practical advantages of the proposed method. The authors should provide empirical evidence demonstrating how their approach improves upon these established alternatives.

  3. Experimental Limitations.

(1) Insufficient experimental comparisons and validation. Although the paper includes internal control experiments, it lacks comparisons with established retrieval baselines. Furthermore, the evaluation is limited to a single dataset, leaving the generalizability of the proposed method to other datasets unverified.

(2) Insufficient justification of the efficiency evaluation metric. The paper does not sufficiently justify relying on candidate-scoring compression as its primary efficiency metric. Although this metric captures the reduction in scoring workload, it does not directly establish end-to-end speedup. To substantiate the efficiency claims, the authors should compare latency, throughput, and memory usage against optimized baselines at matched retrieval quality.

  1. Presentation issues The paper is overly verbose and repetitive, making it difficult for readers to identify its main technical contributions. Several points, such as the distinction between the proposed method and autoregressive Semantic-ID generation, as well as the mechanism for enforcing hard constraints, are repeatedly emphasized across multiple sections without providing substantial additional insights. Furthermore, much of the discussion focuses on reiterating methodological details and experimental results, with insufficient synthesis of the key findings and their broader implications.
Questions for authors

See the Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.