acceptodds
Under review as a conference paper at ICLR 2027

Autonomous Fashion Outfit Composition via Unified Aesthetic Foresight Model

Abstract

Fashion Outfit Composition (FOC) requires sequentially assembling fashion items into a stylistically cohesive ensemble. Existing works struggle to model this step-by-step process effectively, primarily because they fail to jointly optimize the two critical capabilities required for FOC: intermediate outfit value evaluation and complementary item prediction. This structural disconnect, compounded by the severe sparsity of step-wise aesthetic signals, leaves them without a mechanism to autonomously determine when to stop the composition process. To address these challenges, we introduce the Unified Aesthetic Foresight Model (UAFM), which seamlessly unifies both capabilities into a dual-head architecture over a shared backbone. Crucially, we formulate FOC as a deterministic Markov Decision Process and optimize a value head via a post-decision state temporal difference (TD) objective to recursively backpropagate sparse terminal aesthetic rewards to intermediate states. This enables the model to accurately estimate the potential of a partial outfit evolving into a compatible ensemble. Consequently, UAFM can compute step-wise marginal aesthetic gains for dynamic termination. Furthermore, this unified design strictly aligns aesthetic evaluation and complementary item prediction within a shared representation manifold, intrinsically regularizing the combinatorial search space. Extensive experiments on the Polyvore-Outfits dataset demonstrate that UAFM establishes a new state-of-the-art across both FOC and conventional fashion tasks. Extensive ablation studies further confirm that our post-decision state TD formulation provides the aesthetic value prediction necessary for autonomous dynamic termination, while validating the synergistic benefits of our unified dual-head architecture.

Primary area:
reinforcement learning
Keywords:
Fashion Outfit Composition, Reinforcement Learning, Temporal Difference Learning, Markov Decision Process, Aesthetic Foresight, Dynamic Termination, Post-Decision State

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YLndOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tackles the task of "generative" Fashion Outfit Composition (FOC), where a model "sequentially" selects compatible garments from a large library and determines when an outfit is complete. It proposes the Unified Aesthetic Foresight Model (UAFM), which combines a policy head for complementary item retrieval and a value head for evaluating partial outfits. A key contribution is the use of Temporal Difference (TD) learning to propagate sparse aesthetic rewards from complete, human-curated outfits to intermediate partial outfits, enabling the model to estimate marginal aesthetic gains and decide when to stop adding items.

Experiments on Polyvore-Outfits show improvements over adapted composition baselines in aesthetic quality, category consistency, and human preference, as well as competitive results on conventional compatibility prediction and fill-in-the-blank retrieval benchmarks. The ablation studies further suggest that TD-based value learning and the shared dual-head architecture benefit sequential composition.

However, while the reported experiments suggest that the framework is effective for outfit composition, it remains unclear whether the learned stopping behavior reflects genuine aesthetic completeness or primarily captures the construction patterns of human-curated outfits.

Critical strengths

This is the first attempt of a "generative" outfit composition work and experiment results show its effectiveness. Comparisons with existing work on compatibility scoring is also solid.

Critical weaknesses

1. The training supervision does not necessarily reflect aesthetic completeness.

The paper assumes that intermediate subsets of human-curated outfits are aesthetically incomplete and should be further expanded. However, a partial outfit (e.g., a shirt, trousers, and shoes) may already be aesthetically complete, even if the original curated outfit contains additional accessories. The dataset provides supervision on which items belong to a curated outfit, but not on whether adding another item would actually improve its aesthetic quality.

Consequently, it is unclear whether the proposed Temporal Difference (TD) learning framework captures genuine aesthetic saturation, as claimed, or primarily learns the outfit construction patterns and length distribution of the training dataset. This is particularly consequential because autonomous termination based on aesthetic saturation is one of the paper's central contributions.

2. The ablation study does not isolate the contribution of TD learning from partial-outfit supervision.

The key ablation in Section 3.3.1 compares BCE trained exclusively on complete outfits against TD learning that additionally supervises intermediate partial outfits. This changes both the training objective and the states receiving supervision, making it difficult to attribute the observed improvements specifically to TD learning.

A straightforward missing baseline is BCE trained on both complete and partial outfits, keeping the architecture and other training settings unchanged. Without this comparison, the experiments do not establish whether the proposed TD formulation provides benefits beyond simply exposing the model to partial outfits during training. This weakens the experimental support for the paper's main methodological contribution.

Questions for authors

Questions raised in above section.

2 comments

We sincerely thank the reviewer for acknowledging our work as the first attempt at generative outfit composition, as well as for your insightful observations regarding aesthetic completeness and the isolation of our training objectives. Below are our detailed responses.

Response to Weakness 1

UAFM learns genuine aesthetic saturation through the tension between a constant step penalty and a capped terminal reward, intrinsically penalizing over-composition without requiring explicit intermediate labels. Overcoming the lack of intermediate dataset supervision (no signal on whether adding another item would actually improve its aesthetic quality) is precisely the motivation for our MDP formulation and post-decision state TD learning paradigm.

To explain why UAFM learns genuine aesthetic saturation rather than merely memorizing dataset length distributions, consider how our TD formulation governs both training and inference: During training, positive trajectories terminate with a fixed reward (y=+1.0y = +1.0), while intermediate steps incur a compositional cost (r=−0.1r = -0.1). As an outfit approaches completeness, the marginal value increment naturally diminishes, capping at 1.01.0. Conversely, negative outfit trajectory yield a terminal reward toward −1.0-1.0. Consequently, during inference, when the Policy Head retrieves a candidate item anexta_{next} to form the augmented state St+1a=Sta∪{anext}S_{t+1}^a = S_t^a \cup \{a_\mathrm{next}\}, two scenarios naturally emerge:

  1. Scenario A (Redundant/Clashing Addition): The value head predicts a sharp decline in expected value based on learned negative trajectories.
  2. Scenario B (Harmless but Unnecessary Addition): Adding a non-essential item to an already complete core outfit cannot generate sufficient aesthetic gain to offset the step cost (Length Regularization). Consequently, v^(St+1a)\hat{v}(S_{t+1}^a) fails to yield a marginal gain over v^(Sta)\hat{v}(S_t^a).

In both cases, the marginal gain drops below the termination threshold, halting composition. This is validated by Figure 4(b), which shows value predictions plateauing as aesthetic saturation is reached for human-curated outfits. We will clarify this theoretical connection in the revised methodology.

Response to Weakness 2

To explicitly address your concern regarding the isolation of our training objectives, we have provided the complete ablation results—including the exact missing BCE-with-partial baseline you suggested—in our Global Comment. These results reveals that applying a static BCE loss to intermediate states actively degrades downstream performance (e.g., R@10 drops from 11.77%11.77\% to 8.96%8.96\%) because coarse binary labels corrupt the decision boundary. Our TD formulation substantially outperforms BCE-with-partial across both generative composition (Category Alignment jumps from 59.51%59.51\% to 78.20%78.20\%, and Category Duplication drops from 14.09%14.09\% to 1.29%1.29\%) and downstream tasks (FITB rises from 66.68%66.68\% to 70.84%70.84\%, and R@10 from 8.96%8.96\% to 12.05%12.05\%). This confirms that the performance gains are fundamentally driven by the step-wise credit assignment of the TD formulation, not merely from exposing the model to intermediate partial outfits.

Dear Reviewer,

The discussion period is nearing its end, and we haven’t yet seen a reply from you. We encourage you to respond to the authors’ comments, especially if you have further questions about points that may affect your assessment.

Thank you for taking the time to engage in the discussion.

Best,

Your AC

Reviewer_BkpbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces UAFM, a unified model that retrieves complementary fashion items and evaluates partial outfits to decide when to stop. Experiments on Polyvore-Outfits show improved composition quality and retrieval performance, with support from human evaluations.

Critical strengths
  1. The research problem is interesting and practical. Sequential outfit composition with autonomous stopping is a worthwhile research direction.
  2. A coherent method. Jointly modeling next-item retrieval and partial-outfit value aligns well with sequential composition.
  3. The results are promising Improvements across composition, compatibility prediction, and retrieval are further supported by human preferences.
Critical weaknesses

See questions

Questions for authors
  1. Do the performance gains mainly come from TD learning or from additional supervision on partial outfits? Could you compare against directly predicting the final return?
  2. Greedy search achieves nearly the same aesthetic score as beam search at much lower latency. Does the model mainly rely on local compatibility rather than long-term planning?
  3. How do you define Completeness? An outfit can work with or without additional accessories. How does the model distinguish “already complete” from “still worth extending”?
2 comments

We sincerely thank the reviewer for recognizing the value of our research direction and the coherence of our method. Below are our responses to your questions.

Response to Question 1:

The performance gains stem from introducing TD learning, which significantly outperforms both direct final-return prediction (MC) and partial-outfit BCE supervision. Please refer to our Global Comment for the full quantitative ablations isolating these mechanisms.

Response to Question 2:

We respectfully clarify that the model does not rely merely on local compatibility; rather, long-term planning is intrinsically internalized within the Value Head. When the model relies strictly on local compatibility (e.g., the Policy-Only baseline in Table 4), Category Alignment drops severely from 78.99%78.99\% to 53.86%53.86\%, while Visual Redundancy surges to 2.78%2.78\%. This proves that local compatibility alone is insufficient for global outfit harmony.

Because our TD objective trains the Value Head to predict the long-term aesthetic potential of the full trajectory, evaluating a post-decision state in Value-Greedy inherently captures long-term planning. Therefore, Value-Greedy matching Beam Search does not imply that long-term planning is absent; rather, it directly proves that our TD-trained Value Head has successfully compiled long-term planning into aesthetic foresight value estimates.

Response to Question 3:

We define "completeness" dynamically as the point of aesthetic saturation, where the marginal aesthetic gain of adding another item falls below a predefined threshold. During the composition process, our model continuously monitors these step-wise marginal aesthetic gains to decide when to stop.

We entirely agree with your observation that fashion lacks an objective, rigid boundary for completeness. Because of this inherent ambiguity, we utilize the human-curated outfits in the dataset as an empirical proxy to validate this dynamic termination approach in Section 3.3.1. As demonstrated in Figure 4(b), when our Value Head evaluates these human-curated outfits step-by-step, the predicted aesthetic scores naturally converge and plateau as they reach aesthetic saturation and UAFM also achieves a highly reliable natural stop rate of 96.2%96.2\% (Line 417). Driven by this robust empirical observation, we specifically adopted this stopping mechanism—halting composition whenever the maximum predicted value improvement of the next augmented state yields only negligible gains (Lines 251-252)—to strictly align with our dynamic definition of completeness.

Dear Reviewer,

The discussion period is nearing its end, and we haven’t yet seen a reply from you. We encourage you to respond to the authors’ comments, especially if you have further questions about points that may affect your assessment.

Thank you for taking the time to engage in the discussion.

Best,

Your AC

Reviewer_6XQvOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the Unified Aesthetic Foresight Model (UAFM) for autonomous Fashion Outfit Composition. UAFM formulates outfit composition as a deterministic Markov Decision Process and employs a shared Transformer with two heads: a value head trained using Temporal Difference learning to estimate the aesthetic potential of partial outfits, and a policy head trained using contrastive learning to retrieve complementary items. The model enables dynamic termination based on marginal aesthetic gains. Experiments on Polyvore-Outfits demonstrate improvements in outfit composition quality, compatibility prediction, and item retrieval, supported by ablation studies.

Critical strengths

None

Critical weaknesses
  1. Insufficient validation of dynamic termination.Dynamic termination is a central contribution, but the main FOC comparison in Table 1 constrains the generated outfit length to match the ground truth. Furthermore, the reported 96.2% natural stop rate counts premature termination as successful, while the average generated outfit length (3.61) is below the ground-truth average (4.13). Therefore, the experiments do not convincingly demonstrate that the model can accurately determine when an outfit is complete.

  2. Limited methodological novelty. The proposed framework mainly combines established techniques, including a shared Transformer, dual-task heads, contrastive learning, and TD learning. The post-decision state formulation closely resembles standard value estimation under deterministic transitions. Although the ablations show improvements over BCE training, comparisons with simpler intermediate-state supervision or alternative value estimation methods are missing, making it difficult to establish the necessity and novelty of the proposed TD formulation.

Questions for authors

1.Can the authors evaluate dynamic termination without ground-truth outfit length constraints and report premature stopping and over-composition rates?

2.What advantages does the proposed TD objective offer over simpler intermediate-state supervision methods?

3.Can the authors clarify whether the qualitative figures show genuine, unedited model outputs or AI-generated illustrations, and provide the original experimental outputs for verification?

3 comments

We sincerely thank the reviewer for the constructive questions regarding our dynamic termination mechanism and the effectiveness of TD; below are our responses to your concerns.

Response to Weakness 1 & Question 1

Why Table 1 uses GT length: In Table 1, ground-truth (GT) length is provided strictly as a controlled variable to isolate item-selection quality. Existing baselines fundamentally lack any termination mechanism. If evaluated without length constraints, these baselines would not dynamically stop; instead, they would trivially expand every outfit until hitting an arbitrary, hardcoded max_len cap. That setting would make baseline scores artificially sensitive to the choice of max_len and force severe over-composition. In fact, providing oracle GT length in Table 1 grants baselines a favorable advantage, yet UAFM still outperforms all of them across all aesthetic and structural metric.

Unconstrained Dynamic Termination Evaluation: The Experiment 1 of Section 3.3.1 (L410-419) is specifically designed to evaluate whether our model genuinely achieves dynamic termination. It operates completely without GT length constraint rather than Table 1. Expanding this exact unconstrained ablation with the additional metrics yields the results below:

ObjectiveAvg LengthPremature Stop RateOver-Composition RateExact Match Rate
BCE-complete-only5.546.2% ± 0.5%77.2% ± 0.3%16.6% ± 0.3%
BCE-with-partial5.1911.3% ± 0.2%67.5% ± 0.8%21.2% ± 0.7%
Monte Carlo5.2415.3% ± 0.1%69.1% ± 0.4%15.7% ± 0.3%
TD Loss (Ours)3.6151.7% ± 0.3%21.0% ± 0.3%27.3% ± 0.5%
  • BCE-complete-only is the standard binary cross-entropy supervising the value head strictly on complete outfits.
  • BCE-with-partial: BCE loss extended to evaluate both complete outfits and intermediate partial states as static binary targets.
  • Monte Carlo: Updating intermediate state values via full trajectory Monte Carlo empirical returns without temporal bootstrapping.
  • TD Loss (Ours): Our proposed post-decision state temporal difference objective incorporating recursive updates.

As shown, our proposed TD formulation reduces the unbounded Over-Composition Rate (from 67.5%67.5\% to 21.0%21.0\%) and achieves the highest Exact Match Rate (27.3%27.3\%). This proves that UAFM’s dynamic termination reliably recognizes reasonable aesthetic boundaries, whereas the intermediate-state BCE supervision or MC variant largely fails to prevent unbounded composition.

While UAFM exhibits a higher "premature" stop rate relative to the absolute GT length, we respectfully argue that this is a feature of its aesthetic restraint, not a failure. It's worthnoting that fashion lacks rigid completion boundaries; an outfit can work perfectly well with or without optional accessories (which artificially inflate the GT dataset average length to 4.134.13). In generative FOC, over-composition—such as retrieving duplicate shoes or redundant skirts (see Figure 6 third row)—is highly detrimental to user experience.

Response to Weakness 2 & Question 2

Our TD formulation is necessary; simpler intermediate supervision degrades both FOC and fashion discriminative task performance. As shown in the Global Comment table, simply assigning binary labels to incomplete states (BCE-with-partial) corrupts the model's decision boundary, which degrades downstream discriminative accuracy (e.g., FITB drops from 70.41%70.41\% to 66.68%66.68\%, R@10 drops from 11.77%11.77\% to 8.96%8.96\%). In contrast, our TD objective accurately quantifies partial aesthetic potential, yielding substantial improvements in both generative composition (VAS 3.603.60 vs. 3.403.40, Category Alignment 78.20%78.20\% vs. 59.51%59.51\%) and downstream retrieval (FITB 70.84%70.84\% vs. 66.68%66.68\%).

Response to Question 3

We confirm that all qualitative figures in Figure 6 are genuine, unedited model outputs constructed from the polyvore-outfit dataset images. UAFM directly queries existing item IDs (item_idxs) from the dataset's test split. The figures are rendered programmatically without any post-processing. To provide complete transparency, the exact raw JSON prediction outputs for all five UAFM ensembles in Figure 6 are provided below. Every item_idxs strictly maps to a verifiable image file.

[
  {
    "item_idxs": [212969, 204097, 201977, 188828, 245432],
    "category": ["tops", "bottoms", "shoes", "bags", "sunglasses"],
    "description": [
      "floral embroidered faux suede tank",
      "gap women tiered maxi skirt",
      "espadrilles with lacing",
      "iris ink leather shoulder bag",
      "miu round sunglasses"
    ]
  },
  {
    "item_idxs": [201093, 214456, 34430, 153157],
    "category": ["shoes", "bags", "all-body", "sunglasses"],
    "description": [
      "faux leather cross straps mini",
      "metal bar chain tassel crossbody",
      "white lace short sleeve slim",
      "crossbar mirrored butterfly sunglasses"
    ]
  },
  {
    "item_idxs": [72888, 129071, 143064, 60567],
    "category": ["outerwear", "tops", "shoes", "bags"],
    "description": [
      "paul smith womens sherbet pink",
      "navabi dusky-pink plus size ruffle",
      "saint laurent women 140mm candy",
      "rebecca minkoff dusty pink marlowe"
    ]
  },
  {
    "item_idxs": [73991, 86297, 229133, 51033, 130460],
    "category": ["outerwear", "bags", "shoes", "bottoms", "sunglasses"],
    "description": [
      "boutique moschino two-tone ribbed cotton",
      "jeremy scott flower shoulder bag",
      "giuseppe zanotti design sandals",
      "fausto puglisi mini skirt",
      "retro p3 round color mirrored"
    ]
  },
  {
    "item_idxs": [220402, 170147, 231736],
    "category": ["bottoms", "tops", "shoes"],
    "description": [
      "frame womens le grand garcon",
      "casual spaghetti strap sweater tank",
      "converse opticalwt all star ox"
    ]
  }
]

Dear Reviewer,

The discussion period is nearing its end, and we haven’t yet seen a reply from you. We encourage you to respond to the authors’ comments, especially if you have further questions about points that may affect your assessment.

Thank you for taking the time to engage in the discussion.

Best,

Your AC

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Comment on Isolating the Contribution of Post-Decision State TD Learning

We thank all reviewers for their insightful feedback. The primary performance gains of UAFM stem fundamentally from the temporal regularization of our TD learning mechanism, rather than mere exposure to intermediate partial outfits. To explicitly isolate the contribution of the TD formulation and address the shared questions regarding alternative supervisions, we provide two additional ablation baselines to Table 3:

  1. BCE-with-partial: Applies standard binary BCE loss to intermediate partial outfits to test if simple intermediate supervision suffices.
  2. MC (Monte Carlo): Trains the value head to directly predict the final return (the mathematical sum of the discounted terminal reward and step penalties, YtMC=∑k=tT−2γk−tr+γT−1−tyY_t^{MC} = \sum_{k=t}^{T-2} \gamma^{k-t} r + \gamma^{T-1-t} y) using MSE loss (bypassing TD bootstrapping).
ArchitectureVASCate. Align %Cate. Dup. %↓Vis. Red. %↓AUCFITBR@10R@30R@50
Separate3.27 ± 0.0267.73 ± 0.2918.01 ± 0.326.75 ± 0.220.8468.5710.0919.7226.25
BCE-complete-only3.33 ± 0.0255.23 ± 0.2715.89 ± 0.113.70 ± 0.070.9670.4111.7720.1029.03
BCE-with-partial3.40± 0.0659.51 ± 0.2014.09 ± 0.123.44 ± 0.050.9266.688.9617.7023.73
MC3.53± 0.0361.27± 0.298.65 ± 0.151.64 ± 0.070.9268.5710.5820.0126.82
TD (Ours)3.60 ± 0.0278.20 ± 0.491.29 ± 0.070.34 ± 0.040.9370.8412.0522.6329.50

These results demonstrate that the TD credit assignment mechanism is strictly necessary for the observed performance:

  • TD (Ours) vs. BCE-with-partial: Our TD formulation consistently outperforming BCE-with-partial across every generative and discriminative metric. Specifically, exposing the model to intermediate states with coarse binary labels corrupts the decision boundary, thereby degrading downstream discriminative performance (e.g., FITB drops to 66.68%66.68\% and R@10 to 8.96%8.96\%) compared with BCE-complete-only.
  • TD (Ours) vs. MC: TD learning significantly outperforms direct final-return prediction across both generative composition and discriminative metrics. While MC achieves a moderate Category Alignment of 61.27%61.27\% and a Category Duplication rate of 8.65%8.65\%, our TD formulation drastically improves structural cohesion, lifting Category Alignment to 78.20%78.20\% and suppressing duplication to just 1.29%1.29\%. Furthermore, TD yields consistently higher retrieval accuracy, increasing FITB from 68.57%68.57\% to 70.84%70.84\% and R@10 from 10.58%10.58\% to 12.05%12.05\%.

We will include the results and discussions of these two additional variants in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.