acceptodds
Withdrawn from ICLR 2027

TRACE: Trajectory-Routed Multi-Objective Alignment for Flow Matching Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning from image-level rewards has become a standard approach for post-training flow-based text-to-image models. However, most existing methods optimize a single reward, and aligning multiple objectives such as compositional faithfulness, text rendering, and human preference remains challenging. The core difficulty is that multiple objectives conflict when optimized jointly on shared parameters. The difficulty appears along two temporal dimensions: within the denoising trajectory, objectives respond differently across phases; across optimization updates, simultaneous gradients can conflict and sequential updates can overwrite earlier gains. We therefore propose TRACE (Trajectory-Routed Alignment with Chunked Experts), which first estimates a per-phase multi-objective utility tensor via bucket-restricted probes and uses it to determine both where each objective is optimized and where each is protected. At the timestep level, TRACE converts this tensor into closed-form softmax routing weights that reweight the policy surrogate loss, while a timestep-conditioned LoRA provides phase-specific updates through a shared adapter. At the training-step level, TRACE trains focused single-objective chunks and protects idle objectives with profile-weighted anchor losses. Experiments on FLUX.1-dev and SD3.5-M with GenEval, OCR, and PickScore show simultaneous improvements over the base models and the highest reported point estimates among the evaluated multi-objective baselines on all three optimized metrics; CLIPScore and Aesthetic Score also remain above the corresponding base-model values. Code is available at https://anonymous.4open.science/r/TRACE-C427.

Primary area:
generative models
Keywords:
Diffusion Model, Reinforcement Learning, Multi-Objective Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LsjEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

TRACE focuses on multi-objective alignment for text-to-image models, aiming to simultaneously improve compositional generation, text rendering, and visual preference alignment while mitigating gradient conflicts and forgetting across objectives. Training is organized into single-objective chunks, with anchor losses preserving capabilities associated with temporarily inactive objectives. Experiments on FLUX.1-dev and SD3.5-M show simultaneous improvements in GenEval, OCR, and PickScore, demonstrating the effectiveness of the method.

Critical strengths
  1. The paper provides a detailed analysis of multi-objective conflicts by examining both gradient compatibility during joint training and catastrophic forgetting during sequential training. The empirical measurements provide strong evidence supporting these observations.
  2. The paper presents extensive ablation studies that demonstrate the effectiveness of the individual components of TRACE.
  3. The paper provides a thorough analysis of forgetting mitigation. By experimentally comparing different anchor coefficients, the authors demonstrate a reasonable trade-off between retaining previously learned capabilities and adapting to new objectives.
Critical weaknesses
  1. TRACE measures the optimization gains of different objectives at different denoising stages through short probing runs and uses these measurements to derive temporal routing weights. The same weights are then used to impose anchor constraints on temporarily inactive objectives. However, the rationale for using the same temporal routing weights for both optimization and forgetting protection is not sufficiently justified or validated.

  2. The improvements brought by Tc-LoRA may partially stem from its additional parameter capacity. Tc-LoRA introduces a small number of extra parameters compared to standard LoRA. Including a parameter-matched standard LoRA baseline would help address this concern and better isolate the contribution of timestep conditioning.

  3. The evaluation of multi-objective alignment is not sufficiently comprehensive. In particular, the experiments do not adequately demonstrate whether the model can simultaneously satisfy multiple objectives within a single generated image.

Questions for authors
  1. The rationale for using the same temporal routing weights for both optimization and forgetting protection in TRACE is not sufficiently justified or validated.

  2. The improvements brought by Tc-LoRA may partially stem from its additional parameter capacity. Tc-LoRA introduces a small number of extra parameters compared to standard LoRA. Including a parameter-matched standard LoRA baseline would help address this concern.

  3. The evaluation of multi-objective alignment is not sufficiently comprehensive. In particular, the experiments do not adequately demonstrate whether the model can simultaneously satisfy multiple objectives within a single generated image.

If these concerns are adequately addressed, I would consider raising my score.

Reviewer_r15pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TRACE, an RL framework for aligning FM models with multiple rewards. TRACE first profiles how each reward is optimized at different timesteps, and based on this forms timestep weights for each objective. While training, TRACE optimizes each objective independently using a Tc-LORA adapter, while adding anchor losses to prevent forgetting of previous optimized objectives. Across multiple backbones, the paper demonstrates that TRACE beats prior multi-objective baselines over GenEval, OCR, and PickScore.

Critical strengths

1.) I appreciate that code is provided.

2.) The paper proposes a solution for multi-objective alignment, which is fairly important.

3.) Individual components (e.g. Tc-LORA, anchor losses) are well-motivated

Critical weaknesses

1.) The paper does not offer much intuition/explanation on why certain behaviors happen the way they do. Why should different objectives prefer refinement at different timesteps? Why does GenEval benefit most from early steps while OCR/PickScore benefit more from later timesteps? Detailed explanations would provide much greater clarity.

2.) The temporal profiling procedure appears dubious to some degree, first each profiling run is ran for only 50 optimization steps which seems low, and the division into five temporal buckets appears somewhat arbitrary. Are the identified temporal preferences stable across longer profiling runs (e.g. greater optimization steps)? How do these preferences fare against different timestep discretizations (e.g. finer discretization/more steps, resulting in more buckets)?

3.) Does not contain human evaluation (although usually understandable due to cost). However, in this case, as we are optimizing our model over specific evaluation metrics, there is a heavy assumption that these metrics are representative of true image quality and human preferences, which may not always be the case (Kamath et al., 2025; Ba et al., 2025; Mao et al., 2026). While non-optimized metrics such as CLIPScore and Aesthetic Score are indeed given, I hesitate to conclude that they are sufficient. Human evaluation would strengthen quality claims.

Questions for authors

See Critical Weaknesses for relevant questions

Reviewer_kNuyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies multi-objective post-training of flow matching models for text-to-image generation, focusing on conflicts among compositional faithfulness, text rendering, and preference scores. Building on Flow-GRPO, the authors propose TRACE, which organizes optimization along two temporal dimensions.

First, independent objective-by-denoising-phase probes measure both gains on the optimized objective and effects on other objectives. These measurements form a utility tensor that is converted into fixed softmax timestep-routing weights. Second, timestep-conditioned LoRA (Tc-LoRA) represents phase-dependent adaptation. Finally, training alternates between single-objective chunks, while replay-based anchor losses, weighted by each inactive objective's own routing profile, mitigate forgetting.

Experiments cover FLUX.1-dev and Stable Diffusion 3.5 Medium. Among the evaluated multi-objective methods, TRACE achieves the highest point estimates on all three training-aligned metrics: GenEval, OCR, and PickScore. The paper also reports component ablations, comparisons of routing schemes, training dynamics, and inference overhead.

Critical strengths

S1. Measuring both gains on the optimized objective and cross-objective effects provides a concrete multi-objective basis for temporal routing.

Table 1 contains a useful example: the gain on PickScore itself is highest in bucket B2, whereas its net utility is highest in B3 after accounting for effects on GenEval and OCR. Thus, cross-objective effects actually change the routing decision; they are not merely an additional description of single-objective phase selection.

Turning measurable responses into a training-allocation rule is the paper's most compelling technical idea. The probing procedure also offers a practical tool for analyzing multi-objective generative models.

S2. Results are consistent across two backbones and are supported by informative ablations.

Tables 2 and 5 show that TRACE improves all three optimized objectives on both backbones and exceeds the point estimates of the listed multi-objective baselines. In Table 3, removing either policy time routing or anchor protection causes substantial degradation. Comparisons against random, reversed, and hard routing also support the value of the measured soft routing profiles.

These results are more persuasive than final performance numbers alone. Maintaining a single shared adapter is also practically useful for deployment, although Table 8 shows that inference latency still increases by approximately 5.5%-7.1% relative to standard LoRA.

Critical weaknesses

W1. The central claim that the same measurement should guide both optimization and protection has not been sufficiently isolated and validated.

The paper uses the same routing profile, w_j, both for policy optimization when objective j is active and for anchor protection when it is inactive. However, the probes measure the gains and side effects of optimizing objective j's loss in a particular phase. They do not directly measure which phases are most effective for applying constraints that preserve objective j. The correspondence between these two quantities is a plausible hypothesis, but it still requires empirical support.

Table 3 supports the usefulness of time routing and anchor protection individually, but it does not clearly report a control that retains policy routing while replacing only the anchor-routing weights with uniform weights. Nor does it report routing based solely on gains on the optimized objective, S_m. It is therefore difficult to determine whether the improvements specifically depend on modeling cross-objective effects and reusing the resulting profiles for protection, or primarily reflect the benefits of timestep weighting and general replay regularization.

In addition, Table 7 shows that Tc-LoRA improves single-objective training as well, and the authors acknowledge the absence of a parameter-matched control. Consequently, when comparing against baselines that use standard LoRA, the extent to which the gains arise from the multi-objective mechanism rather than a more capable adapter remains unclear. This limits attribution of the paper's central contributions.

W2. The reliability of the reported performance lead and the comparability of computational budgets remain insufficiently established.

The main results report only point estimates, without specifying the number of training seeds or providing variability across runs or test-set confidence intervals. Some key margins are modest: on FLUX.1-dev, the improvements over the strongest multi-objective baseline for each metric are 0.019 on GenEval, 0.011 on OCR, and 0.122 on PickScore. On SD3.5-M, the GenEval margin is 0.010. Without uncertainty estimates, it is difficult to assess whether these advantages are robust.

The computational budget also requires a complete accounting. In addition to 3,000 main-policy updates, TRACE uses 15 x 50 = 750 probe updates. Its normalization constants depend on trained single-objective experts, and each main update also involves anchor computations for inactive objectives. Matching only the number of main-policy updates does not ensure comparable total computational cost. The baseline comparisons mix models trained by the authors with externally released checkpoints, and the appendix does not provide a consolidated account of costs and training settings sufficient to verify comparability.

The current evidence therefore supports the conclusion that TRACE performs well under the reported settings, but does not yet reliably establish its advantage under comparable resource budgets. CLIPScore and Aesthetic Score remaining above the base-model values is also encouraging, but conclusions about general image quality should remain limited to what these automatic metrics capture.

Questions for authors

Q1. Validation of the central mechanism: With policy routing, Tc-LoRA, scheduling, and the training budget held fixed, what happens when the anchor-routing weights for inactive objectives are replaced with uniform weights? How does routing based only on gains on the optimized objective compare with the full routing rule that includes cross-objective effects? These controls would directly affect my assessment of the central contribution: using the same utility tensor to guide both temporal dimensions.

Q2. Clarification of comparison reliability: Please specify the number of random seeds used for the main results and report variability across training runs, together with appropriate test-set uncertainty estimates, for TRACE and the strongest baselines. Please also provide total computational costs, including expert training, probing, replay-based anchoring, and validation, as well as the joint-training baseline's update budget and prompt-mixture procedure. Do the main advantages persist when total budgets are matched and all methods use Tc-LoRA?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.