- Rating
- 1(−1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates whether behavioural signals from eye-tracking data can be used to estimate the reliability of human preference annotations in pairwise comparison tasks.
The authors collect an eye-tracking dataset using Apple Vision Pro, comprising approximately 4,000 annotations across 797 preference comparison tasks. Their analysis identifies two main findings about annotation behaviour:
- Annotators exhibit two types of behaviours: "readers", whose task completion time increases with response length, and "scanners", whose completion time remains relatively constant.
- Visual attention is unevenly distributed across the displayed responses, with annotators spending more time looking at the left response and the upper-left regions of the text.
Given these findings, the authors propose two approaches to weighting preference labels during majority-vote aggregation: (1) the proportion of text visually inspected, estimated from eye-tracking data, and (2) reading speed, estimated from task completion time and response length. Their results show that reading-speed-weighted aggregation achieves the best overall ranking among the evaluated fine-tuning configurations, although the performance differences vary across benchmarks.
Critical strengths
-
I think that the idea of studying additional behavioural signals during preference annotation tasks is interesting and potentially relevant for LLM alignment.
-
If made publicly available, the detailed annotation dataset, together with reproducible code, could be a useful resource for future research.
Critical weaknesses
These are the main weaknesses:
-
I found the paper generally difficult to follow. Several sentences required multiple readings to understand what the authors meant.
-
The paper does not provide links to the experimental code or the collected dataset. This limits the reproducibility of the experiments and prevents other researchers from independently analysing or reusing the dataset.
-
Downstream improvements are not consistently supported by the results. Specifically:
-
All DPO fine-tuning configurations decrease average AlpacaEval performance compared to the original base models. Although DPO does not necessarily guarantee improvements on every benchmark, this raises questions about the effectiveness of the fine-tuning setup and the claimed improvements in alignment.
-
When comparing only the fine-tuned models, the differences between unweighted and weighted preference aggregation are small on Arena Hard and MT-Bench, considering the reported standard errors. Therefore, it is unclear whether the proposed weighting strategies provide meaningful improvements on these benchmarks.
-
-
Throughout the paper, the authors motivate the use of reading behaviour as an indicator of annotation reliability, but they do not directly validate this assumption. Although they evaluate the impact of the proposed weighting strategies on downstream DPO performance, this does not necessarily demonstrate that slower reading or greater text coverage leads to more reliable preference annotations. Nevertheless, in Section 5.1, the authors state that "Our results indicate that coverage and especially reading speed are useful indicators of preference label reliability." This conclusion seems stronger than what the experimental evidence supports.
Questions for authors
Given that DPO fine-tuning generally decreases AlpacaEval performance compared to the original base models, could the authors clarify why this happens? Furthermore, since different hyperparameters are used across fine-tuning configurations, how can we determine whether the observed improvements with reading-speed-weighted aggregation are due to the weighting strategy rather than differences in the fine-tuning setup?
5 comments
To address the questions for authors:
Given that DPO fine-tuning generally decreases AlpacaEval performance compared to the original base models, could the authors clarify why this happens? Furthermore, since different hyperparameters are used across fine-tuning configurations, how can we determine whether the observed improvements with reading-speed-weighted aggregation are due to the weighting strategy rather than differences in the fine-tuning setup?
Thank you for requesting clarifications. As mentioned in the previous comment, some decrease in performance has also been observed in the original HelpSteer2 papers. Notably, we also use different (smaller) models in fine-tuning than the original papers. The hyper-parameters are chosen to have the highest accuracy on the HelpSteer2 validation set (as described on L351 of the manuscript). Hyper-parameters were tuned per setting to reduce the confounding factor of hyper-parameter quality. For each approach, the hyper-parameters were tuned following the same grid search strategy with the same range of hyper-parameter values to choose from. This strategy was selected to make the comparison as fair as possible by giving each approach its best fine tuning configuration.
Additional references:
Liu, W., Zeng, W., He, K., Jiang, Y., & He, J. (2024). What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning. In B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, & Y. Sun (Eds.), International Conference on Learning Representations (Vol. 2024, pp. 22353–22373). https://proceedings.iclr.cc/paper_files/paper/2024/file/6091f2bb355e960600f62566ac0e2862-Paper-Conference.pdf
Wang, Y., Ivison, H., Dasigi, P., Hessel, J., Khot, T., Chandu, K., Wadden, D., MacMillan, K., Smith, N., Beltagy, I., & Hajishirzi, H. (2023). How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, & S. Levine (Eds.), Advances in Neural Information Processing Systems (Vol. 36, pp. 74764–74786). Curran Associates, Inc. https://doi.org/10.52202/075280-3268
Wang, Z., Bukharin, A., Delalleau, O., Egert, D., Shen, G., Zeng, J., Kuchaiev, O., & Dong, Y. (2025). HelpSteer2-Preference: Complementing Ratings with Preferences. In Y. Yue, A. Garg, N. Peng, F. Sha, & R. Yu (Eds.), International Conference on Learning Representations (Vol. 2025, pp. 56548–56575). https://proceedings.iclr.cc/paper_files/paper/2025/file/8e237ec6d3bc86f6d4967e9c7eef37ff-Paper-Conference.pdf
Wang, Z., Dong, Y., Delalleau, O., Zeng, J., Shen, G., Egert, D., Zhang, J. J., Sreedhar, M. N., & Kuchaiev, O. (2024). HelpSteer 2: Open-source dataset for training top-performing reward models. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, & C. Zhang (Eds.), Advances in Neural Information Processing Systems (Vol. 37, pp. 1474–1501). Curran Associates, Inc. https://doi.org/10.52202/079017-0047
Hi, we thank the authors for the rebuttal.
-
On presentation: For example, I personally had to read the second and third sentences of the Introduction multiple times. These are indeed personal opinions, but I understand that “consider” probably means “obtain” or “use,” although this is not immediately clear. Similarly, it is unclear what “better estimates” refers to in the following sentence. I understand that the other reviewers found the presentation clear, so do consider this as a minor point in my final assessment.
-
Regarding reproducibility, I find the response insufficient. I understand that releasing eye-tracking data collected from human participants may require ethical approval. However, the collected dataset is presented as one of the main contributions of the paper, and neither the dataset nor the experimental code is available to reviewers. The description of the data collection protocol is not a substitute for access to the artifacts needed to reproduce the reported results. Furthermore, it is unclear why the experimental code could not be made available independently of the dataset.
-
On the explanation given to downstream improvements: The authors use results from previous work to normalize a weakness in their own experiments, rather than explaining why their results support their scientific claim. I do not find this argument convincing, as it does not address my central criticism regarding the limited improvements on Arena Hard and MT-Bench relative to the reported standard errors.
Overall, I do not find that the rebuttal adequately addresses my main concerns, particularly regarding reproducibility and the experimental evidence supporting the paper's central claims. For these reasons, I have decided to lower my score from 2 to 1.
Thank you for taking the time to provide a review and feedback. Please find below a response to the critical limitations and questions to authors. To address the critical weaknesses:
Thank you for raising this concern. We would appreciate if you can point us to any particular parts of the paper that are difficult to understand and we are happy to elaborate, as the other two reviewers rated the presentation clear.
Release of paper assets is pending paper acceptance and a final review of the data from our review board. However, we have provided sufficient detail about our data collection protocol for others to replicate our study.
Thank you for asking for a clarification. The decrease in performance w.r.t to baseline models on AlpacaEval is also visible in the original HelpSteer2 paper. To illustrate, Llama 3 70B Instruct baseline outperforms all other training configurations on AlpacaEval in the original HelpSteer2 dataset (Wang et al., 2024). On the preference variant (Wang et al., 2025), Regular DPO condition on HelpSteer2 is slightly better than Llama-3.1-70B-Instruct (40.4 with SE=1.66 vs. 38.1 with SE=0.90). We note that even with significantly fewer labels, our dataset tends to match the performance or improve upon the full HelpSteer2 used in fine-tuning. Also, fine-tuning improvements do not necessarily translate to all benchmarks, which may explain some of the mixed results.
Thank you for raising this concern. First of all, we have amended the wording around the conclusion to reflect that we conclude coverage and reading speed to be potentially useful signals in fine-tuning. As preference annotation is a subjective task, gold labels do not exist and other measures like inter-annotator agreement can be equally inconclusive. Therefore, we use the downstream fine-tuning experiments as a way to disentangle whether reading speed and coverage metrics are a useful signal, similar to other papers that investigate quality of datasets (e.g., Liu et al., 2024 and Wang et al., 2023).