- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a task-aware method for selecting DPO and GRPO configurations. It uses early training signals to predict alignment quality and utility retention for different hyperparameters, reducing the need for exhaustive grid search. Experiments on 25 tasks show improved performance and lower computational cost, although the comparison between DPO and GRPO is affected by their different supervision and reward designs.
Critical strengths
- Configuration selection for preference optimization is computationally expensive, and the proposed use of short-horizon training dynamics to reduce the cost of hyperparameter search is potentially useful. The paper also considers the realistic multi-objective trade-off between alignment quality and utility retention, rather than optimizing alignment quality alone.
- The proposed framework is technically coherent within its stated experimental setup. The combination of early training signals, a task encoder, a configuration-invariance objective, and a dual-objective surrogate is clearly described. The experiments include several meaningful baselines and report configuration-transfer performance, Pareto-front recovery, ablations, and compute savings.
Critical weaknesses
-
The motivation for jointly selecting between DPO and GRPO is not sufficiently justified. DPO and GRPO are not merely two alternative algorithms with different hyperparameters. They rely on substantially different supervision pipelines: DPO uses static preference pairs, whereas GRPO requires online generation and a task-dependent reward function. In the experiments, GRPO uses exact-match rewards for reasoning tasks, unit-test rewards for code tasks, and an external reward model for other tasks. Therefore, the comparison conflates algorithm selection with reward construction, supervision format, and training-pipeline design. The paper should either narrow its claim to task-aware hyperparameter transfer within a fixed training pipeline, or explicitly model and evaluate the dependence on reward availability and reward-model quality.
-
The evidence for a transferable, low-dimensional task fingerprint is limited. The study uses only 25 task instances, a single base model, a fixed LoRA setup, a small configuration grid, and largely instance-level rather than source-disjoint transfer. Moreover, the learned representation may encode reward-model identity, method-specific dynamics, or dataset artifacts rather than intrinsic task properties. The paper does not test whether the proposed fingerprint generalizes across unseen reward functions, data sources, base models, or evaluation criteria.
Questions for authors
- DPO and GRPO rely on fundamentally different supervision pipelines. DPO uses static preference pairs, whereas GRPO requires online sampling and a task-dependent reward function. Why is it appropriate to treat them as interchangeable configurations of the same task?
- For reasoning and code tasks, the paper uses exact-match and unit-test rewards, while other tasks use an external reward model. Does this mean that the comparison involves different training signals across methods and task types? How do the authors control for the effect of reward design?
Response to PGyv: within-pipeline transfer and limits of task fingerprints
Thank you for recognizing the framework and for identifying the distinction between configuration transfer and supervision-pipeline design.
Critical Weaknesses:
W1. Joint DPO/GRPO selection conflates algorithm choice with supervision and rewards.
We agree that conditioning on method identity does not control reward quality. Our primary claim will be hyperparameter transfer within a specified pipeline. The response is more precisely R(task,pipeline)(hyperparameters), with the pipeline including the base model, data construction, training method, and reward implementation. Joint selection is a secondary operational choice when both pipelines are already available.
Using the collected configuration runs, we separate method-specific selection:
Within each task-method pair, data construction and reward implementation are fixed while hyperparameters vary. These numerical means support configuration transfer under those conditions. The DPO-only margin over ASHA is small and will not be presented as a large or universally reliable advantage. Joint HM of 81.0 is a separate selection result over both pipelines, not the mean of the two rows. Fixed recipes, adaptive search, and the outcome-informed sampled reference have different costs and information access.
W2. Limited evidence for an intrinsic, universally transferable low-dimensional fingerprint.
The fingerprint summarizes observed dynamics under the tested model and pipeline. Within-method invariance does not establish reward independence or remove all source artifacts. Likewise, the PCA result describes the learned embeddings in this finite collection; it is not a theorem about intrinsic task dimensionality.
We will restrict the evidence to the tested 25 instances, Llama-3-8B, LoRA setup, and candidate space. The main split holds out instances rather than sources. We do not claim demonstrated transfer to unseen rewards, models, or evaluation criteria. The representation is useful insofar as it predicts outcomes under these specified conditions.
Questions For Authors:
Q1. Why treat DPO and GRPO as interchangeable configurations?
They are not interchangeable under identical supervision. The intended use is conditional pipeline selection: given an available preference-pair route and an available reward-based route, predict which configuration meets the chosen alignment/utility trade-off. If only one route is available, selection stays within that method. The separate DPO-only and GRPO-only results above are therefore the primary evidence. We will revise the task definition, abstract, and joint-selection language accordingly.
Q2. Are different training signals involved, and how is reward design controlled?
Yes. GRPO uses exact-match rewards for reasoning, unit-test rewards for code, and Skywork-Reward-Llama-3-8B-v0.2 for the other tasks; DPO uses preference pairs. This prevents interpreting the joint comparison as an isolated algorithm effect.
The within-method comparison holds each task's data/reward pipeline fixed and changes only the configuration-selection strategy. This controls reward design for that narrower comparison. It does not demonstrate robustness to reward substitutions or quality perturbations. We will document pair construction and reward access in the revision and avoid attributing supervision differences to the optimization algorithm. The resulting contribution is short-dynamics-based configuration transfer, with an explicitly conditional cross-pipeline extension.