acceptodds
Withdrawn from ICLR 2027

SAPO: Self-Anchored Preference Optimization for Cold-Start Neural Scheduling

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Constructive neural solvers for scheduling problems (e.g., JSSP, FJSP) face a critical cold-start dilemma: training from scratch lacks an informative reference prior, making standard preference optimization difficult to apply effectively. While recent reference-free methods alleviate the dependency on expert policies, they may exhibit two stage-specific weaknesses: premature convergence associated with deterministic winner selection in early training, and policy drift when no explicit trust region is available during late-stage refinement. To bridge this gap, we propose Self-Anchored Preference Optimization (SAPO), a stage-adaptive framework that dynamically constructs the missing prior from the training trajectory itself. SAPO introduces a dual-phase mechanism: (1) Distribution-Smoothed Exploration, which samples winners from a decaying top-K pool to broaden early winner supervision; and (2) Self-Anchored Refinement, which utilizes the historical-best policy as a moving anchor to provide an anchor-relative signal during refinement. Across six JSSP benchmark families, SAPO yields lower mean gaps than BOPO under a matched fixed-checkpoint decoding audit and shows improved stability on the recorded late-stage trajectory, while controlled FJSP/TSP studies provide preliminary evidence of transfer to related constructive tasks without systematic inference overhead.

Primary area:
reinforcement learning
Keywords:
neural combinatorial optimization, preference optimization, reinforcement learning, job shop scheduling

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_UFv8OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Self-Anchored Preference Optimization (SAPO), a two-stage framework for cold-start neural scheduling without pretrained reference policies. SAPO introduces distribution-smoothed exploration, which samples winners from a decaying top-K candidate pool to improve early training diversity, followed by self-anchored refinement, which uses the historical-best policy as a dynamic reference to stabilize later optimization. The method is evaluated on six job-shop scheduling (JSSP) benchmark families, with additional transfer experiments on flexible JSSP and TSP. Results show modest improvements over BOPO in optimality gaps, improved late-stage stability on the recorded training trajectory, and comparable inference efficiency. The authors conclude that stage-adaptive preference optimization offers a practical approach to cold-start neural scheduling.

Critical strengths
  1. SAPO addresses the lack of informative reference policies in neural scheduling through a stage-adaptive framework combining diversified early preference sampling with historical-best policy anchoring. The approach is compatible with existing neural schedulers and does not introduce additional inference overhead.
  2. The paper evaluates SAPO across six JSSP benchmark families under matched architecture and decoding settings with BOPO. The phase-wise ablations, anchor-update comparisons, and FJSP/TSP transfer checks provide meaningful evidence of the framework's practical effectiveness, although the observed improvements are modest.
Critical weaknesses
  1. SAPO primarily combines two established techniques (top-K winner sampling and checkpoint-based reference regularization) into a two-stage training framework. While practical, the approach offers limited conceptual or theoretical novelty beyond their integration, weakening its contribution.
  2. The improvement over BOPO is relatively modest (0.312 percentage points in the JSSP macro optimality gap). Moreover, the reported five-seed evaluation measures decoding variability from fixed checkpoints rather than independent training runs. Consequently, the evidence is insufficient to establish consistent improvements in training performance and stability across random initializations.
Questions for authors
  1. The reported five-seed JSSP results measure decoding variability rather than independent training runs. Is there any existing evidence that SAPO consistently outperforms BOPO across different training seeds? How should the claimed training stability be interpreted given this limitation?
  2. Phase 2 changes both the reference policy and the preference loss formulation. How can the authors distinguish the benefits of historical-best anchoring from those arising from the modified loss and its scaling?
  3. Given that top-K sampling and reference-policy regularization are established techniques, what is the key methodological insight that distinguishes SAPO from a straightforward combination of these components?
Reviewer_QxujOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses cold-start preference optimization for constructive neural scheduling, where no pretrained or expert policy is available as an informative reference. It proposes Self-Anchored Preference Optimization (SAPO), a two-phase framework combining distribution-smoothed exploration through a decaying top-K winner pool with self-anchored refinement using a validation-selected historical-best policy as a moving reference. The main contribution is this stage-adaptive training scheme, which leaves the policy architecture and inference procedure unchanged. The paper evaluates SAPO primarily on six JSSP benchmark families, with auxiliary FJSP/TSP experiments and analyses of anchor-update rules and computational cost. It reports lower mean gaps than BOPO under matched fixed-checkpoint decoding and improved stability on the recorded late-stage training trajectory.

Critical strengths

None

Critical weaknesses

The submitted JSSP loss implementation is inconsistent with Eq. (9): for fixed preference pairs, the historical-best policy does not affect the gradient with respect to the current policy parameters, so the implementation does not realize the self-anchored update described in the paper.

Submitted implementation

In JSSP/sampling.py, SROLoss computes:

ms_factor = ms_l / ms_w
loss_main = -torch.log(
    torch.sigmoid(ms_factor * (loss_w - loss_l))
).mean()

kl_w = loss_w - ref_lp_w
kl_l = loss_l - ref_lp_l
avg_kl = (kl_w.mean() + kl_l.mean()) / 2

return loss_main + beta * avg_kl

Consider one fixed preference pair (x,yw,yl)(x,y_w,y_l). Let ℓθw,ℓθl\ell_\theta^w,\ell_\theta^l denote the current policy's mean selected-action log probabilities along the winner and loser trajectories, and let ℓrefw,ℓrefl\ell_{\mathrm{ref}}^w,\ell_{\mathrm{ref}}^l denote the corresponding frozen reference-policy values. Define

λ=Cx(yl)Cx(yw),dθ=ℓθw−ℓθl.\lambda = \frac{C_x(y_l)}{C_x(y_w)}, \qquad d_\theta = \ell_\theta^w - \ell_\theta^l.

The implemented per-pair loss is

Lcode=−log⁡σ(λdθ)+βc2(ℓθw+ℓθl−ℓrefw−ℓrefl),L_{\mathrm{code}} = -\log\sigma(\lambda d_\theta) + \frac{\beta_c}{2}(\ell_\theta^w + \ell_\theta^l - \ell_{\mathrm{ref}}^w - \ell_{\mathrm{ref}}^l),

where βc\beta_c denotes the code's beta coefficient. For the fixed pair, λ\lambda is constant, and the reference policy is frozen during the update. Therefore,

∇θLcode=−λσ(−λdθ)(∇θℓθw−∇θℓθl)+βc2(∇θℓθw+∇θℓθl).\nabla_\theta L_{\mathrm{code}} = -\lambda\sigma(-\lambda d_\theta)(\nabla_\theta\ell_\theta^w - \nabla_\theta\ell_\theta^l) + \frac{\beta_c}{2}(\nabla_\theta\ell_\theta^w + \nabla_\theta\ell_\theta^l).

The reference-policy values disappear entirely from this gradient. Changing the historical-best policy can change the loss value, but cannot change this gradient for fixed preference pairs. Batch averaging preserves this conclusion.

Objective in Eq. (9)

The paper instead defines an anchor-relative margin using full trajectory log probabilities:

Δanc(t)=log⁡πθ(yw∣x)πanc(t)(yw∣x)−log⁡πθ(yl∣x)πanc(t)(yl∣x).\Delta_{\mathrm{anc}}^{(t)} = \log\frac{\pi_\theta(y_w\mid x)}{\pi_{\mathrm{anc}}^{(t)}(y_w\mid x)} - \log\frac{\pi_\theta(y_l\mid x)}{\pi_{\mathrm{anc}}^{(t)}(y_l\mid x)}.

For one preference pair, the loss is

Lpaper=−log⁡σ(βΔanc(t)),L_{\mathrm{paper}} = -\log\sigma(\beta\Delta_{\mathrm{anc}}^{(t)}),

and its gradient is

∇θLpaper=−βσ(−βΔanc(t))(∇θlog⁡πθ(yw∣x)−∇θlog⁡πθ(yl∣x)).\nabla_\theta L_{\mathrm{paper}} = -\beta\sigma(-\beta\Delta_{\mathrm{anc}}^{(t)})(\nabla_\theta\log\pi_\theta(y_w\mid x) - \nabla_\theta\log\pi_\theta(y_l\mid x)).

Although the anchor is frozen, its log probabilities remain in Δanc(t)\Delta_{\mathrm{anc}}^{(t)} and affect the gradient weight, providing the anchor-relative signal for refinement. This effect is absent from the submitted implementation.

Consequently, experiments based on this implementation cannot validate the central claim that the historical-best policy acts as a moving self-anchor to guide late-stage refinement.

Questions for authors
  1. Why does the submitted SROLoss differ from Eq. (9), with reference-policy terms that do not affect the gradient for fixed preference pairs?
  2. If the reference-policy values do not affect the gradient, how did changing only the anchor-update rule produce the different results in Table 5?
Reviewer_hwMbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SAPO addresses cold-start preference optimization for neural scheduling through two stages: top-K winner sampling for early exploration and a historical-best policy reference for later refinement. It improves JSSP benchmark gaps over BOPO under matched decoding budgets.

Critical strengths
  1. Simple training modifications preserve the architecture and inference procedure.
  2. Matched BOPO comparisons and anchor-update ablations provide useful empirical evidence.
Critical weaknesses
  1. Main JSSP results repeat decoding from fixed checkpoints rather than independent training runs, leaving training robustness uncertain.
  2. The proposed drift-control mechanism is insufficiently validated: anchor-relative preference margins do not directly constrain policy divergence.
Questions for authors
  1. Do the improvements persist across independent training seeds?
  2. Does Phase 2 retain BOPO’s objective-guided scaling? Please specify β\beta and isolate the reference term’s contribution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.