acceptodds
Under review as a conference paper at ICLR 2027

Using eye-tracking to understand and estimate reliability of preference labels

Abstract

Large Language Model alignment methods like DPO rely on availability of high quality preference annotations, where annotators are asked to choose which of typically two responses they prefer. Often, it is assumed that these human-annotated labels are reliable in that they represent preferences well. However, these annotations may not reflect underlying preferences due to factors like labeling errors, task difficulty, biases introduced by the experimental setup, etc. While examining the impact of noisy labels on downstream tasks has gained attention in the literature, less emphasis has been placed in understanding where the noise comes from—how annotators complete the annotation tasks. In this paper, we use eye-tracking to examine how preference annotation is done and collect an eye-tracking dataset EyeTracking-HS2-800 using Apple Vision Pro (including 5 annotations for approximately 800 tasks from the HelpSteer2–Preference dataset completed by 100 participants). Our results point to positional biases in the distribution of visual attention over the samples, and two distinct strategies for completing the annotation tasks emerge: "readers" display an increasing task completion time and “scanners” a relatively constant task completion time with the length of the shown sample. We then use these observations to modulate weight of individual human-annotated preference labels so that more carefully annotated samples get a higher weight (measured by proportion of fixated text and reading speed). By increasing the weight of the "reader" annotations, we recover, and on average even outperform, the performance of all HelpSteer2-Preference labels despite having significantly fewer annotations, outperforming all other training configurations. Based on our results, we provide guidelines for conducting and reporting preference annotation studies. In particular, we suggest incorporating reading speed, or time spent per task, as a suitable alternative to improve LLM alignment in the case of annotator disagreement.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Preference annotation, eye-tracking, LLM fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_4qgVOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates whether behavioural signals from eye-tracking data can be used to estimate the reliability of human preference annotations in pairwise comparison tasks.

The authors collect an eye-tracking dataset using Apple Vision Pro, comprising approximately 4,000 annotations across 797 preference comparison tasks. Their analysis identifies two main findings about annotation behaviour:

  • Annotators exhibit two types of behaviours: "readers", whose task completion time increases with response length, and "scanners", whose completion time remains relatively constant.
  • Visual attention is unevenly distributed across the displayed responses, with annotators spending more time looking at the left response and the upper-left regions of the text.

Given these findings, the authors propose two approaches to weighting preference labels during majority-vote aggregation: (1) the proportion of text visually inspected, estimated from eye-tracking data, and (2) reading speed, estimated from task completion time and response length. Their results show that reading-speed-weighted aggregation achieves the best overall ranking among the evaluated fine-tuning configurations, although the performance differences vary across benchmarks.

Critical strengths
  • I think that the idea of studying additional behavioural signals during preference annotation tasks is interesting and potentially relevant for LLM alignment.

  • If made publicly available, the detailed annotation dataset, together with reproducible code, could be a useful resource for future research.

Critical weaknesses

These are the main weaknesses:

  • I found the paper generally difficult to follow. Several sentences required multiple readings to understand what the authors meant.

  • The paper does not provide links to the experimental code or the collected dataset. This limits the reproducibility of the experiments and prevents other researchers from independently analysing or reusing the dataset.

  • Downstream improvements are not consistently supported by the results. Specifically:

    • All DPO fine-tuning configurations decrease average AlpacaEval performance compared to the original base models. Although DPO does not necessarily guarantee improvements on every benchmark, this raises questions about the effectiveness of the fine-tuning setup and the claimed improvements in alignment.

    • When comparing only the fine-tuned models, the differences between unweighted and weighted preference aggregation are small on Arena Hard and MT-Bench, considering the reported standard errors. Therefore, it is unclear whether the proposed weighting strategies provide meaningful improvements on these benchmarks.

  • Throughout the paper, the authors motivate the use of reading behaviour as an indicator of annotation reliability, but they do not directly validate this assumption. Although they evaluate the impact of the proposed weighting strategies on downstream DPO performance, this does not necessarily demonstrate that slower reading or greater text coverage leads to more reliable preference annotations. Nevertheless, in Section 5.1, the authors state that "Our results indicate that coverage and especially reading speed are useful indicators of preference label reliability." This conclusion seems stronger than what the experimental evidence supports.

Questions for authors

Given that DPO fine-tuning generally decreases AlpacaEval performance compared to the original base models, could the authors clarify why this happens? Furthermore, since different hyperparameters are used across fine-tuning configurations, how can we determine whether the observed improvements with reading-speed-weighted aggregation are due to the weighting strategy rather than differences in the fine-tuning setup?

5 comments

Thank you for taking the time to provide a review and feedback. Please find below a response to the critical limitations and questions to authors. 
To address the critical weaknesses:

I found the paper generally difficult to follow. Several sentences required multiple readings to understand what the authors meant.

Thank you for raising this concern. We would appreciate if you can point us to any particular parts of the paper that are difficult to understand and we are happy to elaborate, as the other two reviewers rated the presentation clear.

The paper does not provide links to the experimental code or the collected dataset. This limits the reproducibility of the experiments and prevents other researchers from independently analysing or reusing the dataset.

Release of paper assets is pending paper acceptance and a final review of the data from our review board. However, we have provided sufficient detail about our data collection protocol for others to replicate our study.

Downstream improvements are not consistently supported by the results. Specifically: All DPO fine-tuning configurations decrease average AlpacaEval performance compared to the original base models. Although DPO does not necessarily guarantee improvements on every benchmark, this raises questions about the effectiveness of the fine-tuning setup and the claimed improvements in alignment. When comparing only the fine-tuned models, the differences between unweighted and weighted preference aggregation are small on Arena Hard and MT-Bench, considering the reported standard errors. Therefore, it is unclear whether the proposed weighting strategies provide meaningful improvements on these benchmarks.

Thank you for asking for a clarification. 
The decrease in performance w.r.t to baseline models on AlpacaEval is also visible in the original HelpSteer2 paper. To illustrate, Llama 3 70B Instruct baseline outperforms all other training configurations on AlpacaEval in the original HelpSteer2 dataset (Wang et al., 2024). On the preference variant (Wang et al., 2025), Regular DPO condition on HelpSteer2 is slightly better than Llama-3.1-70B-Instruct (40.4 with SE=1.66 vs. 38.1 with SE=0.90). We note that even with significantly fewer labels, our dataset tends to match the performance or improve upon the full HelpSteer2 used in fine-tuning. Also, fine-tuning improvements do not necessarily translate to all benchmarks, which may explain some of the mixed results.

Throughout the paper, the authors motivate the use of reading behaviour as an indicator of annotation reliability, but they do not directly validate this assumption. Although they evaluate the impact of the proposed weighting strategies on downstream DPO performance, this does not necessarily demonstrate that slower reading or greater text coverage leads to more reliable preference annotations. Nevertheless, in Section 5.1, the authors state that "Our results indicate that coverage and especially reading speed are useful indicators of preference label reliability." This conclusion seems stronger than what the experimental evidence supports.

Thank you for raising this concern. First of all, we have amended the wording around the conclusion to reflect that we conclude coverage and reading speed to be potentially useful signals in fine-tuning. As preference annotation is a subjective task, gold labels do not exist and other measures like inter-annotator agreement can be equally inconclusive. Therefore, we use the downstream fine-tuning experiments as a way to disentangle whether reading speed and coverage metrics are a useful signal, similar to other papers that investigate quality of datasets (e.g., Liu et al., 2024 and Wang et al., 2023).

To address the questions for authors:

Given that DPO fine-tuning generally decreases AlpacaEval performance compared to the original base models, could the authors clarify why this happens? Furthermore, since different hyperparameters are used across fine-tuning configurations, how can we determine whether the observed improvements with reading-speed-weighted aggregation are due to the weighting strategy rather than differences in the fine-tuning setup?

Thank you for requesting clarifications. As mentioned in the previous comment, some decrease in performance has also been observed in the original HelpSteer2 papers. Notably, we also use different (smaller) models in fine-tuning than the original papers. The hyper-parameters are chosen to have the highest accuracy on the HelpSteer2 validation set (as described on L351 of the manuscript). Hyper-parameters were tuned per setting to reduce the confounding factor of hyper-parameter quality. For each approach, the hyper-parameters were tuned following the same grid search strategy with the same range of hyper-parameter values to choose from. This strategy was selected to make the comparison as fair as possible by giving each approach its best fine tuning configuration.

Additional references: 


Liu, W., Zeng, W., He, K., Jiang, Y., & He, J. (2024). What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning. In B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, & Y. Sun (Eds.), International Conference on Learning Representations (Vol. 2024, pp. 22353–22373). https://proceedings.iclr.cc/paper_files/paper/2024/file/6091f2bb355e960600f62566ac0e2862-Paper-Conference.pdf

Wang, Y., Ivison, H., Dasigi, P., Hessel, J., Khot, T., Chandu, K., Wadden, D., MacMillan, K., Smith, N., Beltagy, I., & Hajishirzi, H. (2023). How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, & S. Levine (Eds.), Advances in Neural Information Processing Systems (Vol. 36, pp. 74764–74786). Curran Associates, Inc. https://doi.org/10.52202/075280-3268

Wang, Z., Bukharin, A., Delalleau, O., Egert, D., Shen, G., Zeng, J., Kuchaiev, O., & Dong, Y. (2025). HelpSteer2-Preference: Complementing Ratings with Preferences. In Y. Yue, A. Garg, N. Peng, F. Sha, & R. Yu (Eds.), International Conference on Learning Representations (Vol. 2025, pp. 56548–56575). https://proceedings.iclr.cc/paper_files/paper/2025/file/8e237ec6d3bc86f6d4967e9c7eef37ff-Paper-Conference.pdf

Wang, Z., Dong, Y., Delalleau, O., Zeng, J., Shen, G., Egert, D., Zhang, J. J., Sreedhar, M. N., & Kuchaiev, O. (2024). HelpSteer 2: Open-source dataset for training top-performing reward models. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, & C. Zhang (Eds.), Advances in Neural Information Processing Systems (Vol. 37, pp. 1474–1501). Curran Associates, Inc. https://doi.org/10.52202/079017-0047

Hi, we thank the authors for the rebuttal.

  1. On presentation: For example, I personally had to read the second and third sentences of the Introduction multiple times. These are indeed personal opinions, but I understand that “consider” probably means “obtain” or “use,” although this is not immediately clear. Similarly, it is unclear what “better estimates” refers to in the following sentence. I understand that the other reviewers found the presentation clear, so do consider this as a minor point in my final assessment.

  2. Regarding reproducibility, I find the response insufficient. I understand that releasing eye-tracking data collected from human participants may require ethical approval. However, the collected dataset is presented as one of the main contributions of the paper, and neither the dataset nor the experimental code is available to reviewers. The description of the data collection protocol is not a substitute for access to the artifacts needed to reproduce the reported results. Furthermore, it is unclear why the experimental code could not be made available independently of the dataset.

  3. On the explanation given to downstream improvements: The authors use results from previous work to normalize a weakness in their own experiments, rather than explaining why their results support their scientific claim. I do not find this argument convincing, as it does not address my central criticism regarding the limited improvements on Arena Hard and MT-Bench relative to the reported standard errors.

Overall, I do not find that the rebuttal adequately addresses my main concerns, particularly regarding reproducibility and the experimental evidence supporting the paper's central claims. For these reasons, I have decided to lower my score from 2 to 1.

Reviewer_D1YrOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper examines eye tracking as a potentially valuable signal during the preference annotation task, since eye tracking captures artifacts of the decision process that can be potentially informative e.g. in understanding annotator disagreements. The major contribution of this paper is an eye tracking dataset, collected from 100 participants performing a preference annotation task with a total of about 4,000 trials. A brief analysis of the eye tracking data itself is performed, and the authors identify two reading patterns of annotators: "scanners" who skim the item before answering, and "readers" who spend a longer time on the item. Experiments are conducted using DPO to fine-tune models using the 800-item dataset, with different weighting schemes to resolve disagreements according to the eye tracking data (word coverage and reading time). Results suggest that reading time is an especially strong weighting scheme.

Critical strengths
  • The most critical strength is the dataset, EYETRACKING-HS2-800. It is a non-trivial, time-consuming, and expensive dataset to collect, and I believe it is valuable to the community. I am only aware of one other dataset, from Lopez-Cardona et al., that contains eye movements over the preference annotation task, so this contribution meaningfully increases our ability to conduct experiments and validations when researching the potential of eye movement data to improve or inform alignment.
Critical weaknesses
  • The details of the experiments presented are unclear, and so it is difficult for me to draw strong conclusions with respect to the relative strength of the proposed weighting schemes. Specifically, with respect to the results presented in Table 1: Can we find a per-model breakdown of these results? How consistent are these aggregated rankings across each individual model, and are there any trends e.g. with respect to model size? Do you repeat the training runs using different seeds and aggregate the results, or is the variance reported purely from the different models used?
  • Similarly, some details of the eye tracking setup are unclear (see 'questions for authors'), and I strongly recommend moving the technical details of classifying a participant as a "reader" or as a "scanner" from the appendix to the main text, so that your results can be more clearly understood and contextualized. (Also, how many participants are "readers" vs "scanners"?)
Questions for authors

Questions:

  • The agreement between your participants and the original HelpSteer2 is weak. Do you have any insight into why this might be (e.g., population preferences drifting over time, differences between your population and the original HelpSteer2 population, etc.)? Would any of these differences influence how we should interpret the results on the reported eval sets?
  • More detail is needed for the eye tracking setup. (1) Vision Pro specs, e.g. What is the sampling rate? Tracking one eye or both? (2) How long are the responses that can fit on the display without scrolling? (3) Is the post-processing of the data done by your own internal software or is it provided by Apple Vision Pro? Will you share your post-processing scripts? (4) What is the fixation filtering procedure exactly, and how did you decide? Specifically, the reported minimum fixation duration of 45ms is unusually short, and no maximum fixation duration is mentioned.

Minor suggestions and notes:

  • The related work is addressing eye tracking datasets and eye tracking in alignment. I think it is missing a short discussion on eye-tracking-based analyses of annotator/reader behavior more broadly, especially as the introduction lists analysis of how annotators approach preference annotation as the first contribution. Some papers to consider: (1) Hindennach, S., Shi, L., & Bulling, A. (2024). Explaining Disagreement in Visual Question Answering Using Eye Tracking., (2) Mitsuda, K., Ryu I., and Takenobu T. (2013). Detecting missing annotation disagreement using eye gaze information., (3) de Langis, K., Walker, W., Le, K. C., & Kang, D. (2026). Tracing how annotators think: Augmenting preference judgments with reading processes.
  • I found the writing to be generally clear and well-organized, but the writing style overall would benefit from more polish. Two quick examples: "completed relatively fast" --> "completed relatively quickly"; "quality of the DPO’ed model" --> "quality of the [DPO model / DPO-fine-tuned model / etc]"
2 comments

Thank you for taking the time to review our paper and for providing feedback. Below, we have addressed the two main critical limitations and questions for authors.

Firstly, to address the critical limitations:

The details of the experiments presented are unclear, and so it is difficult for me to draw strong conclusions with respect to the relative strength of the proposed weighting schemes. Specifically, with respect to the results presented in Table 1: Can we find a per-model breakdown of these results? How consistent are these aggregated rankings across each individual model, and are there any trends e.g. with respect to model size? Do you repeat the training runs using different seeds and aggregate the results, or is the variance reported purely from the different models used?

Thank you for asking for a clarification. The per-model breakdown of the results is presented in Appendix E, Table 5. The rankings tend to vary per model, but particularly in AlpacaEval one of the variants of ET-HS2-800 tends to outperform the original HelpSteer2—preference dataset used in fine-tuning. In terms of the rankings, the model size tends to not make a big difference, however, the larger models tend to outperform the smaller ones as expected.

Table 1 reports standard error across LLMs.

Similarly, some details of the eye tracking setup are unclear (see 'questions for authors'), and I strongly recommend moving the technical details of classifying a participant as a "reader" or as a "scanner" from the appendix to the main text, so that your results can be more clearly understood and contextualized. (Also, how many participants are "readers" vs “scanners"?)

Thank you for this suggestion. We have moved the technical details related to the classification of the participants to the main body. There are 54 scanners and 46 readers in the bins based on the threshold of 4.5 s / 100 px of increased reading time. We note that this cut-off could be done in different ways and this would be interesting to investigate in future work.

To address the questions for authors:

The agreement between your participants and the original HelpSteer2 is weak. Do you have any insight into why this might be (e.g., population preferences drifting over time, differences between your population and the original HelpSteer2 population, etc.)? Would any of these differences influence how we should interpret the results on the reported eval sets?

More detail is needed for the eye tracking setup. (1) Vision Pro specs, e.g. What is the sampling rate? Tracking one eye or both? (2) How long are the responses that can fit on the display without scrolling? (3) Is the post-processing of the data done by your own internal software or is it provided by Apple Vision Pro? Will you share your post-processing scripts? (4) What is the fixation filtering procedure exactly, and how did you decide? Specifically, the reported minimum fixation duration of 45ms is unusually short, and no maximum fixation duration is mentioned.

The related work is addressing eye tracking datasets and eye tracking in alignment. I think it is missing a short discussion on eye-tracking-based analyses of annotator/reader behavior more broadly, especially as the introduction lists analysis of how annotators approach preference annotation as the first contribution. Some papers to consider: (1) Hindennach, S., Shi, L., & Bulling, A. (2024). Explaining Disagreement in Visual Question Answering Using Eye Tracking., (2) Mitsuda, K., Ryu I., and Takenobu T. (2013). Detecting missing annotation disagreement using eye gaze information., (3) de Langis, K., Walker, W., Le, K. C., & Kang, D. (2026). Tracing how annotators think: Augmenting preference judgments with reading processes.

I found the writing to be generally clear and well-organized, but the writing style overall would benefit from more polish. Two quick examples: "completed relatively fast" --> "completed relatively quickly"; "quality of the DPO’ed model" --> "quality of the [DPO model / DPO-fine-tuned model / etc]”

Thank you for these comments. Indeed, we observe differences in the preference labels in our dataset and HelpSteer2—preference. There are some intentional differences in the experiment design of the original HelpSteer2 data collection and ours, most notably that our data collection happened in-person with an Apple Vision Pro headset, while the original was crowd-sourced and labels collected in a screen-based interaction. HelpSteer2 also only collected five labels from tasks with low inter-annotator agreement, while we did this for all tasks. These differences in labels also motivated a direct comparison of our majority vote labels (ET-HS2) with the original ones (HS2-800). Furthermore, due to the differences in labelling agreement, fine-tuning results from the weighted schemes (ET-HS2-coverage and ET-HS2-speed) should be compared to our data (ET-HS2). We observe some improvement on the chosen benchmarks with our ET-HS2 data, and further improvement when using the reading speed / coverage weightings, suggesting they provide additional benefits.

Thank you for requesting clarifications for the experimental setup, presentation and for pointing us to additional references. We will update the manuscript with additional details.

Reviewer_SYg6OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary
  1. The paper collects a useful eye-tracking dataset for LLM preference annotation. Examining how annotators read and compare responses provides information that is usually missing from preference datasets.
  2. The study connects human reading behavior with downstream LLM alignment. The experiments cover several models and compare different label aggregation strategies, making the findings relevant to preference data collection.
Critical strengths
  1. The paper collects a useful eye-tracking dataset for LLM preference annotation. Examining how annotators read and compare responses provides information that is usually missing from preference datasets.
  2. The study connects human reading behavior with downstream LLM alignment. The experiments cover several models and compare different label aggregation strategies, making the findings relevant to preference data collection.
Critical weaknesses
  1. The relationship between reading behavior and label reliability is not sufficiently established. Longer reading times or greater gaze coverage do not necessarily indicate more accurate judgments. They may also reflect task difficulty, reading ability, or uncertainty. The paper lacks independent validation showing that labels from slower readers are more reliable.
  2. The evidence for improved LLM alignment is limited. In Table 1, reading-speed weighting performs better than several other fine-tuning conditions, but does not consistently outperform the original base models. The improvements also vary across benchmarks. Moreover, the paper does not adequately compare the proposed weighting with simpler alternatives, such as completion-time or agreement-based weighting. It is therefore unclear how much benefit comes specifically from the proposed reliability measures.
Questions for authors
  1. What independent evidence shows that slower reading or higher gaze coverage produces more reliable preference labels? How do the authors account for differences in reading ability and task difficulty?
  2. Table 1 shows that reading-speed weighting does not consistently improve performance over the base models. How robust are the reported advantages across models, benchmarks, and training runs?
  3. How does the proposed approach compare with simpler weighting methods based on task completion time, annotator agreement, or random weights?
4 comments

Thank you for taking the time to provide this review. Below, we address the critical limitations and questions for authors.

Firstly, to respond to the main critical limitations:

The relationship between reading behavior and label reliability is not sufficiently established. Longer reading times or greater gaze coverage do not necessarily indicate more accurate judgments. They may also reflect task difficulty, reading ability, or uncertainty. The paper lacks independent validation showing that labels from slower readers are more reliable.

Thank you for raising this concern. Whether behavioral metrics like reading speed (derived from task completion times) and coverage can serve as proxies for label quality is a hypothesis that we investigated through the downstream fine-tuning experiments. A similar approach is used in other papers that try to investigate quality of datasets: different data mixtures are used in fine-tuning and the performance of the fine-tuned models is evaluated on benchmarks (e.g., Liu et al., 2024 and Wang et al., 2023). We recognize that the relationship between behavioral metrics and label quality is not straight-forward, and we have clarified the wording around this assumption.

The evidence for improved LLM alignment is limited. In Table 1, reading-speed weighting performs better than several other fine-tuning conditions, but does not consistently outperform the original base models. The improvements also vary across benchmarks. Moreover, the paper does not adequately compare the proposed weighting with simpler alternatives, such as completion-time or agreement-based weighting. It is therefore unclear how much benefit comes specifically from the proposed reliability measures.

Thank you for asking for this clarification. The reduced performance w.r.t to the base model is also visible in the original paper related to HelpSteer2. In Wang et al. (2024) containing the original HelpSteer2 dataset, Table 4 shows that the base model (Llama 3 70B Instruct) outperforms DPO with HelpSteer2 and PPO with HelpSteer2 in MT Bench and AlpacaEval (though not in mean response length, ArenaHard and Truthful QA). In Wang et al. (2025) containing the HelpSteer2—preference variant, Llama-3.1-70B-Instruct base used in training is slightly worse than the fine-tuned models, though Claude-3-5-Sonnet-20240620 or GPT-4o-2024-05-13 bases outperform Regular DPO and PPO in most benchmarks as indicated by Table 4. Therefore, it appears that the inconsistencies could stem from the HelpSteer2 dataset. We also note that our work is concerned with the utility of coverage and reading speed weightings specifically within fine-tuning, rather than with performance in general.

As for the simpler alternatives, the reading speed weighting is a task completion time metric (see L335 in the manuscript). The weight of reading speed (ET-HS2-speed condition) is computed as follows: 1) the total task completion time is divided by the number of characters to give an estimate of reading speed, and 2) this reading speed is compared to population reading speed, and an inverse taken. Therefore, a higher task completion time gets a higher weighting. Also, we compare our results to a simple majority vote (ET-HS2 condition), which is a version of agreement-based weighting. We interpret that an agreement based weighting could consider IAA of the set of annotations in one task, and use that to weight the importance of that majority vote within the whole task set (=dataset), which is an approach to dataset weighting. This is distinct from the per-sample weighting approach we consider in this paper. Whether agreement based weighting is useful is an interesting question, albeit a different one we set out to answer. If you refer to something else with an agreement based weighting here, we are happy to elaborate.

To answer the questions for authors:

  1. What independent evidence shows that slower reading or higher gaze coverage produces more reliable preference labels? How do the authors account for differences in reading ability and task difficulty?
  2. Table 1 shows that reading-speed weighting does not consistently improve performance over the base models. How robust are the reported advantages across models, benchmarks, and training runs?
  3. How does the proposed approach compare with simpler weighting methods based on task completion time, annotator agreement, or random weights?

Through the downstream fine-tuning evaluation, the results in our paper test whether a task that is completed slower, or looked at more improves preference label quality. The weights are always computed within-task, therefore the task difficulty is stable across the weights. We do not control for reading ability, and we have highlighted this in limitations.

A table showing the per-model breakdown of the results is reported in Appendix E, Table 5.

The reading speed weighting is based on task completion times. An agreement-based weighting would be a sensible choice when evaluating reliability of a task’s majority vote label against all tasks in the set. However, we evaluate reliability of an individual label within task. Please refer to our comments on critical limitation (2) for additional detail.


Additional references:



Liu, W., Zeng, W., He, K., Jiang, Y., & He, J. (2024). What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning. In B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, & Y. Sun (Eds.), International Conference on Learning Representations (Vol. 2024, pp. 22353–22373). https://proceedings.iclr.cc/paper_files/paper/2024/file/6091f2bb355e960600f62566ac0e2862-Paper-Conference.pdf

Wang, Y., Ivison, H., Dasigi, P., Hessel, J., Khot, T., Chandu, K., Wadden, D., MacMillan, K., Smith, N., Beltagy, I., & Hajishirzi, H. (2023). How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, & S. Levine (Eds.), Advances in Neural Information Processing Systems (Vol. 36, pp. 74764–74786). Curran Associates, Inc. https://doi.org/10.52202/075280-3268

Wang, Z., Bukharin, A., Delalleau, O., Egert, D., Shen, G., Zeng, J., Kuchaiev, O., & Dong, Y. (2025). HelpSteer2-Preference: Complementing Ratings with Preferences. In Y. Yue, A. Garg, N. Peng, F. Sha, & R. Yu (Eds.), International Conference on Learning Representations (Vol. 2025, pp. 56548–56575). https://proceedings.iclr.cc/paper_files/paper/2025/file/8e237ec6d3bc86f6d4967e9c7eef37ff-Paper-Conference.pdf

Wang, Z., Dong, Y., Delalleau, O., Zeng, J., Shen, G., Egert, D., Zhang, J. J., Sreedhar, M. N., & Kuchaiev, O. (2024). HelpSteer 2: Open-source dataset for training top-performing reward models. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, & C. Zhang (Eds.), Advances in Neural Information Processing Systems (Vol. 37, pp. 1474–1501). Curran Associates, Inc. https://doi.org/10.52202/079017-0047

Thank you for the response. It clarifies some details, but it fails to address my central concerns. I will maintain my original score.

1. The central premise remains untested

The paper's premise is that reading behavior indicates label reliability, and the response offers no evidence for it. Using downstream fine-tuning as the test is circular, because downstream performance is exactly what the labels are meant to explain. Without an independent check against expert or gold labels, the paper cannot claim that slower or more complete reading produces more reliable labels.

The response even makes the problem worse. By clarifying that reading speed is just task completion time divided by text length, it shows that the best-performing condition needs no eye-tracking at all, while the eye-tracking measure, coverage, performs worse. The results do not support the title's claim that eye-tracking can be used to estimate label reliability.

2. The fine-tuning doesnot help at all

Every fine-tuned condition in Table 1 performs worse than the base model, and the "best" condition is simply the one that degrades least. This pattern is equally consistent with the weights weakening the training signal. Pointing to HelpSteer2's own results does not resolve this. The basic control, random weights, is still missing, and no results across training runs are reported.

With differences mostly within the standard errors on Arena-Hard and MT-Bench and only four models, the claim that reading-speed weighting "outperforms all other training configurations" is not supported. The argument that agreement-based weighting applies only at the dataset level is also incorrect. Annotators can be weighted per annotation by their agreement with others on other tasks.

Thank you for commenting on our rebuttal.

To reply to point 1:

We agree that some of the wording around the central claim could be improved. However, one of the points we are precisely trying to make is that for subjective tasks there aren't expert or gold labels that we can compare against. Therefore, we explore whether using behavioral metrics that reflect reliability can be useful in fine-tuning.

Similarly, the title could be rephrased. That said, we argue that the result that reading speed is a potentially more useful weighting metric than eye-tracking is promising: we may be able to get the benefits of an eye-tracking metric---that is far more expensive to collect---by using reading speed. We hardly see this as a result that “makes the problem worse”. In contrast, this observation could have interesting practical implications for fine-tuning work.

To reply to point 2:

Thank you for the comments. We view 1) the performance of the fine-tuned models vs. base and 2) performing the experiments with random weights as two separate matters. We emphasise that degradation of performance is expected with HelpSteer2 as shown by the original papers and we use fewer labels in fine-tuning (800 vs. thousands of labels). However, it is a good idea to make the comparison to random weights and we will consider adding that.

With respect to the wording around the results, we can amend this to refer to "fine-tuning configurations" rather than "training configurations".

We are happy to consider an agreement based weighting. However, pointing to our contributions, we investigate how people perform preference annotation tasks and how this information could be used in fine-tuning. We already compare against majority vote labels, which use a type of agreement based weighting. If another type of agreement based weighting would perform better, it would answer a different hypothesis than what we set out to investigate.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.