acceptodds
Under review as a conference paper at ICLR 2027

Task-Aware Configuration Selection for Preference Optimization via Early Training Dynamics

Abstract

Aligning large language models through Direct Preference Optimization (DPO) or Group Relative Policy Optimization (GRPO) is highly sensitive to configurations such as the regularization strength , learning rate, and training horizon, yet the best configuration varies substantially across tasks. Rather than treating this variability as a tuning nuisance, we ask whether it reflects shared structure in the space of alignment tasks. We formalize each task by its alignment response function, the map from configurations to the joint outcome of alignment quality and utility retention, and posit the Alignment Response Hypothesis: response functions can be represented by low-dimensional task fingerprints extracted from the first hundred training steps. We operationalize the hypothesis with three alignment-dynamics signals processed by an iTransformer encoder under a configuration-invariance objective, together with a dual-objective surrogate that recovers the Pareto front of a new task. Across 25 task instances spanning helpfulness, safety, reasoning, summarization, and dialogue, the learned fingerprints exhibit low-dimensional organization, support task-specific DPO/GRPO selection, and recover 91.2% of the sampled-reference hypervolume with an inverted generational distance of 0.032. The selected configurations reach 81.0 harmonic mean, 96.4% of the sampled-grid oracle, while preserving utility; after meta-training is amortized, inference plus one full run uses 15.1 less compute than per-task grid search.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM alignment, preference optimization, GRPO

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PGyvOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a task-aware method for selecting DPO and GRPO configurations. It uses early training signals to predict alignment quality and utility retention for different hyperparameters, reducing the need for exhaustive grid search. Experiments on 25 tasks show improved performance and lower computational cost, although the comparison between DPO and GRPO is affected by their different supervision and reward designs.

Critical strengths
  1. Configuration selection for preference optimization is computationally expensive, and the proposed use of short-horizon training dynamics to reduce the cost of hyperparameter search is potentially useful. The paper also considers the realistic multi-objective trade-off between alignment quality and utility retention, rather than optimizing alignment quality alone.
  2. The proposed framework is technically coherent within its stated experimental setup. The combination of early training signals, a task encoder, a configuration-invariance objective, and a dual-objective surrogate is clearly described. The experiments include several meaningful baselines and report configuration-transfer performance, Pareto-front recovery, ablations, and compute savings.
Critical weaknesses
  1. The motivation for jointly selecting between DPO and GRPO is not sufficiently justified. DPO and GRPO are not merely two alternative algorithms with different hyperparameters. They rely on substantially different supervision pipelines: DPO uses static preference pairs, whereas GRPO requires online generation and a task-dependent reward function. In the experiments, GRPO uses exact-match rewards for reasoning tasks, unit-test rewards for code tasks, and an external reward model for other tasks. Therefore, the comparison conflates algorithm selection with reward construction, supervision format, and training-pipeline design. The paper should either narrow its claim to task-aware hyperparameter transfer within a fixed training pipeline, or explicitly model and evaluate the dependence on reward availability and reward-model quality.

  2. The evidence for a transferable, low-dimensional task fingerprint is limited. The study uses only 25 task instances, a single base model, a fixed LoRA setup, a small configuration grid, and largely instance-level rather than source-disjoint transfer. Moreover, the learned representation may encode reward-model identity, method-specific dynamics, or dataset artifacts rather than intrinsic task properties. The paper does not test whether the proposed fingerprint generalizes across unseen reward functions, data sources, base models, or evaluation criteria.

Questions for authors
  1. DPO and GRPO rely on fundamentally different supervision pipelines. DPO uses static preference pairs, whereas GRPO requires online sampling and a task-dependent reward function. Why is it appropriate to treat them as interchangeable configurations of the same task?
  2. For reasoning and code tasks, the paper uses exact-match and unit-test rewards, while other tasks use an external reward model. Does this mean that the comparison involves different training signals across methods and task types? How do the authors control for the effect of reward design?
1 comment
AuthorsOpenReview ↗

Response to PGyv: within-pipeline transfer and limits of task fingerprints

Thank you for recognizing the framework and for identifying the distinction between configuration transfer and supervision-pipeline design.

Critical Weaknesses:

W1. Joint DPO/GRPO selection conflates algorithm choice with supervision and rewards.

We agree that conditioning on method identity does not control reward quality. Our primary claim will be hyperparameter transfer within a specified pipeline. The response is more precisely R(task,pipeline)(hyperparameters), with the pipeline including the base model, data construction, training method, and reward implementation. Joint selection is a secondary operational choice when both pipelines are already available.

Using the collected configuration runs, we separate method-specific selection:

HMFixedCategory transferASHAOursSampled reference
DPO only74.479.180.080.283.5
GRPO only73.678.478.980.783.9

Within each task-method pair, data construction and reward implementation are fixed while hyperparameters vary. These numerical means support configuration transfer under those conditions. The DPO-only margin over ASHA is small and will not be presented as a large or universally reliable advantage. Joint HM of 81.0 is a separate selection result over both pipelines, not the mean of the two rows. Fixed recipes, adaptive search, and the outcome-informed sampled reference have different costs and information access.

W2. Limited evidence for an intrinsic, universally transferable low-dimensional fingerprint.

The fingerprint summarizes observed dynamics under the tested model and pipeline. Within-method invariance does not establish reward independence or remove all source artifacts. Likewise, the PCA result describes the learned embeddings in this finite collection; it is not a theorem about intrinsic task dimensionality.

We will restrict the evidence to the tested 25 instances, Llama-3-8B, LoRA setup, and candidate space. The main split holds out instances rather than sources. We do not claim demonstrated transfer to unseen rewards, models, or evaluation criteria. The representation is useful insofar as it predicts outcomes under these specified conditions.

Questions For Authors:

Q1. Why treat DPO and GRPO as interchangeable configurations?

They are not interchangeable under identical supervision. The intended use is conditional pipeline selection: given an available preference-pair route and an available reward-based route, predict which configuration meets the chosen alignment/utility trade-off. If only one route is available, selection stays within that method. The separate DPO-only and GRPO-only results above are therefore the primary evidence. We will revise the task definition, abstract, and joint-selection language accordingly.

Q2. Are different training signals involved, and how is reward design controlled?

Yes. GRPO uses exact-match rewards for reasoning, unit-test rewards for code, and Skywork-Reward-Llama-3-8B-v0.2 for the other tasks; DPO uses preference pairs. This prevents interpreting the joint comparison as an isolated algorithm effect.

The within-method comparison holds each task's data/reward pipeline fixed and changes only the configuration-selection strategy. This controls reward design for that narrower comparison. It does not demonstrate robustness to reward substitutions or quality perturbations. We will document pair construction and reward access in the revision and avoid attributing supervision differences to the optimization algorithm. The resulting contribution is short-dynamics-based configuration transfer, with an explicitly conditional cross-pipeline extension.

Reviewer_iPBuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Direct Preference Optimization and Group Relative Policy Optimization have been widely adopted to align the behavior of LLMs to a given preference; however, one drawback is that they are sensitive to their hyperparameter choices. This paper proposes the Alignment Response Hypothesis: the first hundreds of training steps contain useful information about the task fingerprint, which can be used to predict the alignment and utility score across different hyperparameters after full fine-tuning. They train an iTransformer to encode three fine-tuning signals into a task fingerprint vector, then train a surrogate to condition on this task fingerprint and the hyperparameter choices to predict the full fine-tuning scores. Training on 17 datasets and testing on 8 held-out datasets, they show that their proposed hyperparameter selection method outperforms other baselines.

Critical strengths
  • The paper discusses an important direction for LLM alignment: as full fine-tuning becomes more and more expensive, developing a promising way to perform hyperparameter selection before full fine-tuning is critical.
  • The paper includes multiple analyses to support the proposed method.
Critical weaknesses
  • As the authors state in the limitations, only one model is not enough to support the proposed claim, as different pretrained models may exhibit different behaviors.
  • Some critical experimental settings are missing:
    • How do the authors split the 17 and 8 task instances? How much do the two sets overlap? Will the choices of this split affect the performance?
    • The authors state in line 225 that they use "low, medium, and high regularization at a conservative learning rate" for performing three probes without giving the exact settings.
    • The authors state in line 208 that "Practitioners then select an operating point matching their preferred trade-off," without specifying what the setting is for Table1.
  • An obvious baseline Fixed Recipe for GRPO is missing.
  • Figure 1 contradicts their proposed method, as DPO and GRPO should not impose cross-method invariance.
  • The results of Figure 4 (b) and (c) are hard to interpret, as the meanings of the labels on the x-axis are not explained.
  • The current Out-of-distribution expeiments is too limited, as the transferability to OOD datasets should be a critical aspect of the proposed method. The authors should expand the combinations of in-distribution v.s. out-of-distribution datasets and compare against all the established baselines.
Questions for authors
  • For the results in Table 1, why use the win rate against the reference model but not the win rate against the compared baselines?
  • Since DPO and GRPO adopt different training targets and require different dataset settings, why mix the two together when conducting the analysis? Also, what are the configurations does the proposed method choose, and the grid search find for each task instance?
  • Can we use the best hyperparameter configs found during meta-training to apply to a new dataset of the same type? (Helpfulness, safety...)
  • While DPO is a more supervised method, GRPO requires the model to generate samples, which may induce lots of randomness. How does the proposed method ensure that the first hundreds of fine-tuning steps induce a similar task fingerprint across seeds?
4 comments
AuthorsOpenReview ↗

Response to iPBu (1/2): critical weaknesses

Thank you for identifying the details needed to assess the method. This part follows your weaknesses in order; Part 2 answers your questions.

Critical Weaknesses:

W1. Only one pretrained model is evaluated.

Our evidence is conditional on Llama-3-8B and the tested LoRA setup. Because early dynamics depend on the base model, a predictor trained on one model should not be assumed to transfer unchanged to another. We will restrict the claim to task transfer under a fixed model and specified training pipelines.

W2. Missing experimental settings.

W2.1. Task split, overlap, and split dependence. The 17/8 split is category-stratified using seed 2027 and is not source-disjoint. Instances contain 2,000 training pairs and 200 held-out prompts. The exact-match audit found no evaluation examples duplicated in meta-training. We acknowledge that sample separation does not establish source independence or robustness to every split. The held-out labels are UltraFeedback, BeaverTails, SafeRLHF, PRM800K, TL;DR, CNN/DM, HH-Help, and Arena. We will provide the full construction/membership manifest in the revision; our generalization claim concerns these evaluated instances.

W2.2. Conservative probes. Each method uses three 100-optimizer-step probes, initialized from the same base checkpoint. DPO beta values are 0.05/0.10/0.20; GRPO beta values are 0.01/0.04/0.10. Both use learning rate 1e-6 and AdamW with betas (0.9,0.999). DPO uses batches of 16 sequences, with maximum length 512; GRPO uses 16 prompts and eight rollouts per prompt at temperature 1.0, without additional reward shaping. Fingerprints are averaged within method.

W2.3. Operating point in Table 1. The proposed selector maximizes predicted HM among the 16 sampled candidates, subject to predicted utility at least 93.0%; ties favor higher predicted utility, then lower beta. It uses surrogate predictions rather than final target-task outcomes. The sampled reference maximizes observed HM over the same candidate set. It is an outcome-informed reference, not an oracle over the continuous configuration space. The selection constraint and reference definition will be stated separately.

W3. Missing fixed GRPO baseline.

Fixed GRPO uses beta=0.04, learning rate 1e-5, and three epochs, chosen from meta-training defaults without target-task outcomes. Its HM is 73.6. Method-specific comparisons are:

HMFixedCategory transferASHAOursSampled reference
DPO only74.479.180.080.283.5
GRPO only73.678.478.980.783.9

These are numerical means. In particular, the 0.2-point DPO-only margin over ASHA warrants a modest interpretation. Joint selection reaches 81.0; it is not the average of the method-specific rows. Search costs and one-run costs will be distinguished rather than described as identical for every baseline and the sampled reference.

W4. Figure 1 appears to impose cross-method invariance.

The loss in Section 4.2 is within the same task and method. DPO and GRPO fingerprints are not forced to coincide; averaging is method-specific and the surrogate receives method identity. Figure 1 will show this scope explicitly.

W5. Figure 4(b,c) labels are unclear.

In the submitted Figure 4(b), the x-axis labels are Align, KL, and Ours: two direct 100-step selection signals and our learned selector. Blue bars show rank correlation on the left axis; orange bars show HM regret on the right. Figure 4(c) compares Fixed, In-dist, OOD, and Oracle, with Align/Util/HM as the grouped bars. Its category-shift protocol is helpfulness+safety meta-training and reasoning/code evaluation. These definitions will be added explicitly to the caption.

W6. Limited OOD evaluation.

The reported 76.0 HM concerns the specific helpfulness/safety-to-reasoning/code shift. It does not establish arbitrary OOD, source-disjoint, or cross-model transfer. We will present it as a targeted stress test and keep the primary claim within the evaluated task family.

AuthorsOpenReview ↗

Response to iPBu (2/2): questions for authors

Questions For Authors:

Q1. Why use a common reference rather than win rates against each baseline?

A shared reference gives all rows the same opponent, prompts, and judging criterion. Higher scores against that reference do not guarantee direct head-to-head wins, and we will not imply otherwise. As an additional fixed-checkpoint safety check, unsafe-response rates are 14.2% for ours, 21.4% for Fixed Recipe, and 17.9% for ASHA; over-refusal is 6.4%, 5.8%, and 7.1%. These outcomes are reported separately from preference win rate.

Q2. Why combine DPO and GRPO, and which configurations are selected?

Q2(a). Supervision differences. The operational decision is among already available pipelines. DPO requires preference pairs; GRPO requires online generation and a specified reward. The joint comparison does not identify reward-independent algorithm superiority. We therefore emphasize the within-method comparisons in W3 and condition cross-method selection on the availability of both pipelines.

Q2(b). Selected configurations. The majority selections across the three seeds are:

TaskOurs: method / beta / LR / epochsSampled reference
UltraFeedbackDPO / 0.10 / 5e-6 / 3GRPO / 0.04 / 1e-5 / 3
BeaverTailsGRPO / 0.10 / 1e-5 / 3GRPO / 0.04 / 5e-5 / 3
SafeRLHFDPO / 0.10 / 1e-5 / 3DPO / 0.50 / 1e-5 / 3
PRM800KGRPO / 0.04 / 1e-5 / 3GRPO / 0.10 / 5e-5 / 3
TL;DRDPO / 0.10 / 1e-5 / 3DPO / 0.20 / 5e-5 / 3
CNN/DMDPO / 0.20 / 1e-5 / 3DPO / 0.50 / 5e-5 / 3
HH-HelpDPO / 0.10 / 5e-6 / 3DPO / 0.20 / 1e-5 / 3
ArenaGRPO / 0.04 / 1e-5 / 3GRPO / 0.10 / 1e-5 / 3

For BeaverTails, seeds 0 and 2 select the listed GRPO configuration, while seed 1 selects DPO with beta=0.20. The table summarizes selections, whereas performance is averaged over task-seed runs. The reference uses final outcomes and cannot be deployed as a prediction-only selector.

Q3. Can a good same-category configuration simply be transferred?

Yes. We added category transfer: choose the configuration using meta-training instances of the same category, then apply it without target-task outcome access. Its joint HM is 79.6 versus 81.0 for our method. The paired task differences are 0.3, 0.9, 1.6, 0.2, 2.4, 0.8, 1.1, and 3.7 points, averaging approximately 1.4. This supports incremental value beyond category membership in the evaluated instances, rather than implying that category transfer is ineffective.

Q4. How stable are early GRPO fingerprints across seeds?

We repeated probes for PRM800K, CodeContests, and Arena using five rollout seeds at a fixed initialization. Mean within-task cosine similarity was 0.91, versus 0.42 between tasks; top-choice agreement was 14/15 across the repeated probes. This is a limited empirical stability check, not a seed-invariance guarantee. Appendix B's averaging result requires independent, unbiased noise assumptions; practical validity rests on selection stability and measured outcomes.

I thank the authors for their thorough response. However, two of my main concerns remain:

  • The current experimental results are based only on a single model (Llama-3-8B); whether the proposed method can be applied to other LLMs remains unanswered.
  • The OOD experiments are still too limited; whether the proposed method can outperform all the established baselines on other combinations of in-distribution v.s. out-of-distribution datasets remains unanswered.

Without further evidence addressing these concerns, I will maintain my score of 2.

AuthorsOpenReview ↗

Follow-up on model coverage and OOD evaluation

Thank you for following up. We understand that you would like to see results on another base model and more ID/OOD combinations.

We have no additional results on those settings to report in this discussion. The revised paper now states the scope more precisely: task-level configuration transfer with a fixed base model and specified training pipelines. It also describes the reported category-shift experiment as one particular OOD setting.

Your comments helped us separate the evidence for this task-transfer setting from the broader question of transfer across models and distributions. We appreciate the careful reading.

Reviewer_hT9uOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies task-aware hyperparameter selection for DPO and GRPO. It proposes using early training dynamics to construct task fingerprints, which are then used by a dual-objective surrogate to predict alignment quality and utility retention across configurations.

The framework combines three training-dynamics signals, an iTransformer encoder, and a within-method invariance objective. Experiments on 25 task instances, with 17 used for meta-training and 8 held out for evaluation, show improved configuration selection over several baselines. The method achieves an HM score of 81.0, compared with 84.0 for the sampled-grid reference, while reducing amortized per-task search compute.

Critical strengths
  1. Practical relevance. The paper addresses a practical problem in LLM alignment, as DPO and GRPO are sensitive to hyperparameter choices. Using early training dynamics to reduce the cost of configuration search is a useful direction.
  2. Reasonable method design and promising results. The proposed framework combines three training dynamics signals with an invariance objective to learn task representations. The ablation studies show that the main components contribute to the performance. The method achieves an HM score of 81.0, compared with 74.4 for the fixed recipe and 84.0 for grid search, while reducing the marginal computational cost after meta-training.
Critical weaknesses
  1. Evaluation metrics. The paper uses GPT-4 instruction-following win rate to evaluate alignment quality across different tasks. However, this metric may not fully reflect task-specific objectives, especially for safety and reasoning tasks. It is unclear whether the selected configurations also improve actual safety or reasoning accuracy. This limits how strongly the results support the claimed task-aware alignment performance.
  2. Limited generalization and unclear task construction. The experiments use only Llama-3-8B and 8 held-out task instances. The train/test split is not source-disjoint, and the paper does not clearly explain how the 25 task instances are constructed or provide the exact split. This makes it difficult to assess how well the method generalizes to genuinely new tasks or data sources.
Questions for authors
  1. Evaluation metrics. Could the authors clarify whether the selected configurations also perform well on task-specific metrics, such as safety evaluation and reasoning accuracy? Since the paper uses GPT-4 instruction-following win rate for all tasks, it would be helpful to know whether the improvements are consistent with these task-specific objectives.
  2. Task construction and generalization. Could the authors provide more details on how the 25 task instances were constructed and how the 17 training and 8 test instances were selected? In particular, are there any overlapping samples between instances from the same data source? This would help assess the independence of the test tasks and the reliability of the generalization results.
1 comment
AuthorsOpenReview ↗

Response to hT9u: task-specific evaluation and scope of transfer

Thank you for recognizing the practical value of early-dynamics-based configuration selection. We address each point below.

Critical Weaknesses:

W1. The common judge may not reflect safety and reasoning objectives.

We agree that instruction-following win rate is not a substitute for task-specific evaluation. We evaluated the already selected checkpoints without reselecting configurations on the additional metrics. On 1,000 held-out BeaverTails prompts, unsafe-response rates were 21.4% for Fixed Recipe, 17.9% for ASHA, and 14.2% for our method. On 500 benign prompts, over-refusal rates were 5.8%, 7.1%, and 6.4%, respectively. Decoding was greedy with one response per prompt. The safety classifier had agreement kappa=0.86 against 300 human labels.

Unsafe-response rates decrease by 7.2 percentage points versus Fixed Recipe and 3.7 points versus ASHA. Over-refusal increases by 0.6 points versus Fixed Recipe and decreases by 0.7 points versus ASHA. We report this trade-off explicitly rather than claiming uniform improvement on both safety measures. For reasoning, the main reported alignment outcome remains the specified preference-judge score; we restrict the claim accordingly instead of treating it as reasoning accuracy.

W2. Limited generalization and unclear task construction.

The 17/8 split is category-stratified at the task-instance level, not source-disjoint. An instance is a task-specific pool constructed from a public source, with 2,000 training pairs and 200 held-out prompts. The exact-match audit found no evaluation examples duplicated in meta-training; shared sources can nevertheless introduce related distributions. We therefore limit the main claim to held-out task-instance transfer under Llama-3-8B, rank-16 LoRA, and the tested configuration space. It does not establish transfer across unseen sources, base models, or rewards. We will add the complete instance manifest and construction rules to the revised manuscript.

Questions For Authors:

Q1. Do selected configurations also perform well on task-specific metrics?

The safety check above supports lower unsafe-response rates relative to both Fixed Recipe and ASHA, with the benign-prompt trade-off reported separately. We do not use the common judge to claim independently established reasoning accuracy. Configuration selection was fixed before the safety evaluation.

Q2. How were the 25 instances and split constructed, and do same-source samples overlap?

The eight held-out instance labels are UltraFeedback, BeaverTails, SafeRLHF, PRM800K, TL;DR, CNN/DM, HH-Help, and Arena. Instances are defined by source and task-specific construction; split seed 2027 was used for the category-stratified 17/8 assignment. Our exact-match audit found no cross-split duplicate examples. Sample separation should not be confused with source separation: the current experiment tests new instances, and the manuscript will explicitly state that boundary. A full source/instance membership table is needed for reproducibility and will be included in the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.