acceptodds
Under review as a conference paper at ICLR 2027

When Richer Preferences Do Not Help: Rubric Redundancy, Pair Selection, and Training Exposure in Scientific QA

Abstract

Fine-grained rubrics and dense preference pairs appear to offer richer supervision for aligning scientific-domain large language models (LLMs), but nominal supervision richness need not translate into non-redundant usable preference information. We study this question on 4,000 titanium-alloy reverse-design questions, each associated with four scored candidate answers and task-specific rubric vectors. In the audited single-checkpoint matrix, clean-start binary and six-pair graded DPO both display approximately 92.6 mean generated-answer score at one-decimal precision. Rubric dimensions are strongly correlated for several reasoning types, and at q50 rubric-absolute and scalar-gap pair selection overlap substantially (Jaccard 0.943), although the scalar score is itself aggregated from the rubric dimensions. Across three training seeds, rubric-aware q50 filtering does not establish a reliable advantage over scalar or random selection. Moreover, a seed-42 truncated full-data probe reaches 92.85 after approximately 9.7k preference-pair exposures, compared with 92.31 after a full pass over 19.2k pairs, showing that one-epoch filtered-versus-full comparisons change training exposure together with pair composition. Together, these results suggest that, in this controlled setting, nominal supervision richness can be a poor proxy for non-redundant reference information; preference-signal efficiency should therefore be analyzed through feedback redundancy, pair composition, and training exposure.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language model alignment, Direct preference optimization, Preference data selection, Preference signal efficiency, Rubric redundancy, Training exposure, Scientific-domain question answering

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BsxpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether incorporating additional answer comparison pairs and refining scoring dimensions can provide more effective supervision for preference optimization in scientific question-answering (QA). Drawing on existing literature, the authors utilized a large language model (LLM) to construct a QA dataset for the inverse design of titanium alloys. The dataset comprises 4,000 questions, each featuring four candidate answers, an overall score, and task-specific sub-scores. Using the Qwen3.5-9B model, the authors compared "binary DPO"—constructed from pairs with the largest score gaps—against "dense DPO," which utilizes six pairs per question. Training pairs were selected based on criteria including rubric-based sub-scores, scalar score differences, and random sampling. Automated evaluation across a standardized set of 413 questions revealed that both "clean-start" binary DPO and dense DPO achieved a score of approximately 92.6. Certain task-specific sub-scores showed high correlation, and the Jaccard similarity between selection sets derived from rubric-based differences versus scalar score differences was 0.943 (at the 50th percentile). Across three experimental runs, no consistent advantage for rubric-based selection was established. Furthermore, an analysis of training data volume indicated that reducing the total amount of processed data could still yield performance comparable to that of filtered training sets. The primary contribution of this work lies in its empirical diagnosis of issues related to scoring redundancy, the composition of preference pairs, and the confounding effects of training data volume.

Critical strengths

1.The research addresses a problem of practical value. By analyzing supervision density, scoring structures, and filtering strategies—all based on the same question-answering resource—the authors provide a useful reference for constructing preference data in scientific domains.

  1. The experimental analysis is rigorous, and the reporting of results is transparent. The authors examine potential confounding factors, such as the evaluation process and the volume of training data, while clearly delineating the evidentiary boundaries regarding the use of multiple random seeds and exploratory findings.
Critical weaknesses

1.There is a lack of independent verification for the data and evaluations. Although the authors traced the origins of the literature and conducted manual spot checks, they did not provide a comprehensive report on the verification results for the preference rankings. Furthermore, there is no independent assessment of the new responses generated by the models; the primary conclusions remain heavily reliant on automated scoring systems.

2.The experimental controls are not sufficiently matched. The control conditions simultaneously alter question coverage, training volume, and update configurations. Moreover, the number of repetitions for key comparisons is limited, making it difficult to disentangle the effects of the supervisory structure from those of the training configuration.

Questions for authors

1.What are the sample size for manual verification, the backgrounds of the reviewers, and the accuracy of their preference rankings? Is it possible to provide independent evaluations of the newly generated model responses?

2.Do the results across multiple random seeds—while maintaining consistent initialization, a fixed batch size, matched question coverage, and an identical training budget—still support the primary observations?

3.What is the scope of accessibility for the data, as well as the configurations used for generation and scoring? For restricted datasets, can an auditable subset be provided to allow for independent verification of the preference labels and key findings?

2 comments
AuthorsOpenReview ↗

Human Validation, Experimental Controls, and Reproducibility

We have revised Section 4.2 and added Appendix G to make the available evidence and its limitations more explicit.

1. Human verification: sample size, evaluator background, and agreement

The underlying TitaniumQA resource underwent a construction-stage human validation study.

We sampled 100 questions from each of four reasoning types, selecting the highest- and lowest-scored original candidate answers for each question. This produced 400 questions and 800 answer-level instances.

Three evaluators with materials-science backgrounds independently rated the anonymized answers on a 0–100 scale. The mean of their ratings served as the human reference.

Appendix G.5 now reports the results:

Reasoning typePearsonQWKMAEn
Causal0.8820.8814.59200
What-if0.8390.8234.22200
Pathway0.7920.7396.11200
Mitigation0.7460.7046.51200
Overall0.8220.7965.36800

These statistics compare the construction-stage fused LLM scores with mean human ratings. They provide evidence of human–LLM scoring agreement on the original candidate answers.

We also wish to clarify their scope. Because the subset intentionally contrasts high- and low-scored answers, these statistics should not be generalized to the full range of answer quality. They are not inter-rater agreement estimates or a separately measured pairwise preference-ranking accuracy.

Independent expert evaluation of newly generated model outputs remains unavailable. Such evaluation would require substantial additional specialist effort across many open-ended scientific responses. Our downstream experiments instead employ a frozen rubric-based evaluator to maintain a consistent scoring scale, while acknowledging that this does not remove systematic evaluator bias.

2. Matched-budget experiments and multiple seeds

To address the concern about changing question coverage and optimization budgets, we added a supplementary source-grouped pair-density experiment (Section 5.5, Table 6, Appendix G.1–G.2).

The one-pair random and six-pair conditions share 2,394 training questions, the same SFT initialization, effective batch size 16, and nominal optimization hyperparameters.

They process 14,364 and 14,223 pair exposures over 900 and 889 updates, respectively.

Across two training seeds, one-pair random obtains 93.21/93.02, compared with 92.48/92.83 for six-pair training.

The seed-42 comparison also replicates in direction on a second, previously unused 786-question held-out split (93.30 versus 92.64).

These are approximately matched controls rather than a fully factorial experiment. The two-seed results provide an additional robustness check but are insufficient to establish method-level statistical equivalence.

3. Source-overlap sensitivity and auditability

We additionally examined the effect of source-grouped versus question-level splitting (Appendix G.3).

Under the question-level split, 95.1% of test questions have a primary reference source represented in training; under source grouping, the corresponding overlap is zero.

Using matched training-question counts and task proportions, the corrected DPO scores were 90.39 for source grouping and 90.14 for question-level splitting.

We observed no clear overall inflation associated with reference-source reuse in this single-seed comparison. However, the two conditions also differ in test-question composition and source coverage, so this result does not prove the absence of all forms of leakage.

The new source-grouped train/test/held-out sets contain 186/87/54 primary reference sources, respectively, with no overlap between their primary reference identifiers. We explicitly acknowledge that this does not rule out overlap in auxiliary retrieved evidence or the evaluator's training annotations.

4. Data availability and reproducibility

The complete structured dataset remains subject to project-specific release restrictions.

To support reproducibility within those constraints, the revised manuscript documents the data schema, source-grouped splitting protocol, pair-construction rules, optimization configurations, evaluation procedure, and aggregate audit statistics. The Reproducibility Statement also describes the anonymized supplementary code package containing releasable training, evaluation, analysis, and configuration files.

Releasing actual data instances, including a sampled auditable subset, would require the applicable project approval. We therefore distinguish available implementation and audit materials from restricted underlying records rather than promising unrestricted data access.

We hope the added human-validation statistics, controlled pair-density experiments, source-overlap analysis, and expanded reproducibility documentation address the reviewer's concerns while making the remaining limitations transparent.

Thanks for the detailed response. The human validation numbers (800 instances, Pearson 0.822, QWK 0.796) are welcome, though the rebuttal itself notes the subset was built to contrast high and low answers, so they do not speak to ranking accuracy across the full quality range. The matched-budget runs help, but two seeds and a 900 versus 889 update gap leave them, in the authors' own words, short of statistical equivalence. Independent evaluation of newly generated answers is still missing. On data, no auditable subset is offered pending project approval, which limits external verification of the preference labels. These core concerns survive the rebuttal.

Reviewer_aFSnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether richer preference supervision provides additional usable information for scientific QA alignment. Using 4,000 titanium-alloy reverse-design questions with four rubric-scored answers each, it compares binary and six-pair DPO, audits rubric redundancy, and evaluates rubric-aware pair selection against scalar-gap and random baselines. Its main contributions are an empirical analysis of supervision richness, a characterization of rubric correlations and selection overlap, and an examination of training-exposure confounds. Validation includes a common 413-question evaluation with a frozen domain-adapted scorer, three-seed q50 selection experiments, and single-seed exposure and ablation probes. The results do not establish a reliable advantage from denser pairing or rubric-aware selection and suggest that apparent filtering gains must be interpreted jointly with training exposure, although the evidence remains limited to one domain and model backbone without independent human evaluation of generated outputs.

Critical strengths

The paper asks a useful practical question: whether collecting finer-grained feedback or constructing more preference pairs actually improves alignment. The rubric audit, scalar-gap and random controls, and three-seed selection experiments provide a useful examination of this question. I also appreciate the care taken in interpreting the results: the authors distinguish a failure to establish an advantage from evidence of equivalence, and explicitly acknowledge the limitations of their exposure controls.

Critical weaknesses
  1. The experiments do not sufficiently isolate the factors central to the paper’s conclusions. The binary-versus-graded comparison changes question coverage, pair composition, and training exposure simultaneously. The exposure probes then also change effective batch size, update count, and learning-rate schedules. These experiments identify plausible confounds, but do not resolve them. This leaves the central contribution incomplete: we learn that richer supervision has not shown a consistent benefit in these runs, but gain limited evidence about why.
  2. The main findings depend on an evaluator whose reliability for comparing these model outputs is not established. All reported comparisons rely on a domain-adapted ScoreLLM linked to the preference-scoring pipeline. Its strong agreement with held-out scoring annotations is encouraging, but does not establish that it reliably captures differences in the scientific quality of newly generated answers. Without an independent check, it is difficult to tell whether the relatively small differences—and apparent lack of improvement—reflect model quality or limitations of the evaluator.
  3. Some work like AIR (arXiv:2504.03612) finds that simple point-wise scoring outperforms more elaborate annotation strategies, including fine-grained guidelines. How do you distinguish rubric redundancy from noise or bias introduced by the rubric-based annotation process as an explanation for your results? The correlation and selection-overlap analyses establish similarity between signals, but do you have evidence that helps separate these explanations?
Questions for authors
  1. Regarding the experimental controls: Are there existing comparisons of full-data and filtered-data training that hold effective batch size and the learning-rate schedule fixed while matching the number of updates? If so, do they support the same interpretation? This would help determine whether the exposure probes provide evidence beyond showing that the current comparisons are confounded.

  2. Regarding evaluation reliability: How were the ScoreLLM training and held-out scoring sets separated from the generator evaluation questions? Is there any independent assessment of the generated outputs, through expert judgments or a separately constructed evaluator, that supports the reported similarities and differences between methods? Please distinguish such evidence from the human consistency checks performed during dataset construction.

1 comment
AuthorsOpenReview ↗

Training Controls, Evaluation Reliability, and Rubric Redundancy

We agree that the original exposure probes primarily identify confounding factors rather than fully isolating their causal contributions. We have added supplementary controlled experiments and clarified the distinction between rubric redundancy, scoring reliability, and downstream evaluation validity.

1. Better-controlled pair-density experiments

Section 5.5 and Table 6 introduce a source-grouped comparison using a common SFT checkpoint, the same Qwen3.5-9B backbone, effective batch size 16, and consistent nominal optimization settings.

One randomly sampled pair per question is repeated for six epochs (14,364 exposures; 900 updates), whereas the full six-pair condition processes 14,223 valid distinct pairs (889 updates).

Across seeds 42 and 43, one-pair random obtains 93.21/93.02, compared with 92.48/92.83 for six-pair training.

The seed-42 result also retains its direction on a previously unused 786-question source-disjoint held-out split: 93.30 versus 92.64.

These controls substantially reduce the batch-size and training-exposure differences in the original experiments. However, the two conditions still differ in pair repetition and diversity, and the update counts are not exactly identical. We therefore do not claim complete causal identification.

2. Human validation and the limits of the frozen evaluator

An important clarification in the revision is the construction-stage human validation (Section 4.2; Appendix G.5).

We sampled 100 questions from each of the four reasoning types and selected the highest- and lowest-scored original candidate answers, yielding 800 answer instances. Three evaluators with materials-science backgrounds independently assessed anonymized answers on a 0–100 scale.

The construction-stage fused LLM scores achieved Pearson correlation 0.822, quadratic weighted kappa 0.796, and MAE 5.36 against the mean human ratings.

This is substantive human-agreement evidence for the original scoring protocol. However, it is not an independent human evaluation of the downstream models' newly generated answers, nor does it directly validate the frozen ScoreLLM on those outputs.

For large-scale downstream comparisons, we use a frozen evaluator with identical rubrics and decoding settings to maintain a common quantitative scale. Expert assessment of open-ended scientific reasoning at this scale is resource-intensive, and independent human ratings can introduce additional inter-rater variation.

We nevertheless agree that evaluation consistency is not the same as scientific validity. The frozen evaluator may share biases with the annotation pipeline. We retain this limitation explicitly in Section 7.

Regarding scorer-data separation, the reported performance on held-out scoring annotations establishes agreement with those annotations, not complete independence from the generator evaluation questions. Likewise, the new source-grouped split separates primary reference sources for generator training and evaluation but does not establish complete question- or source-level disjointness for the scorer's own training annotations. We do not claim that this remaining risk has been ruled out.

3. Rubric redundancy versus annotation noise

We appreciate the reference to AIR. We view our study as complementary: AIR examines the effectiveness of preference-data construction strategies, whereas our work examines redundancy and incremental information in the resulting rubric and pair-selection signals.

To investigate scoring reliability beyond correlation and PCA, Appendix G.4 reports a supplementary one-pair selection experiment with identical training budgets and a common initialization.

Selection strategyTest score
Random93.21
Most reliable93.10
Largest total-score gap92.16
Least reliable91.33

The least-reliable pairs have only 42.7% directional agreement between the two construction-stage scoring rounds, compared with 87.8% for random selection. Their downstream score is lower.

Conversely, increasing agreement to 100% does not yield an observed improvement over random selection. The reliable-minus-random difference also retains its direction on the second held-out split (−0.09 versus −0.11 on the original test).

These findings suggest that scoring disagreement may matter for preference learning, while stronger agreement alone does not guarantee better pair selection. However, pair margins, answer sources, and quality profiles also change across selectors.

We therefore do not claim to have isolated rubric redundancy from correlated annotation noise or bias. The revised interpretation is narrower: additional rubric detail and agreement information do not automatically translate into a reliable downstream advantage in the studied setting.

We believe the revised manuscript offers stronger empirical evidence while more clearly identifying what remains unresolved.

Reviewer_7kBCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether richer preference supervision improves scientific QA. It uses 4,000 titanium-alloy questions, each with four candidate answers and rubric scores. The experiments compare binary and six-pair DPO, rubric-based pair selection, and different training exposures. The paper finds strong correlations among several rubric dimensions and substantial overlap between rubric-based and scalar-based selection. Three-seed experiments do not establish a reliable rubric-selection advantage. Additional single-seed experiments show that filtering comparisons also change training exposure.

Critical strengths
  1. The paper addresses a useful experimental issue. Comparing methods for the same number of epochs can change both data composition and training exposure. Reporting pair counts, batch sizes, and update counts makes this problem clear and helps readers assess data-efficiency claims.
  2. The rubric analysis goes beyond aggregate performance. Correlations, PCA, selection overlap, and conflict rates describe what the additional annotations contribute. The pair-gap analysis in Appendix D.3 is particularly useful because it checks whether answer-level correlations overstate redundancy.
Critical weaknesses
  1. The main comparisons do not isolate the value of richer supervision. Binary and graded DPO differ in question coverage, score margins, pair counts, and training exposure (Section 3.1). The exposure probes also change batch size and update counts (Table 5). These results identify confounds but do not resolve them. Moreover, Table 4 gives higher mean scores for both rubric-based selectors than for scalar selection. With only three seeds, the evidence remains inconclusive about whether richer information provides a useful gain.

  2. The evaluation provides limited independent evidence of answer quality. All reported model comparisons use a domain-adapted ScoreLLM linked to the preference-scoring pipeline. Its agreement with held-out scoring annotations does not establish that it can reliably distinguish the newly generated answers. No independent human evaluation is reported. Section 7 also states that several analysis choices were explored on the reporting test set. Together, these limitations weaken confidence that the small score differences reflect reproducible improvements in scientific QA.

Questions for authors
  1. Can you provide a multi-seed comparison with fixed question coverage, batch size, total pair exposure, and learning-rate schedule? For binary versus graded DPO, repeating binary pairs could help match the training budget.

  2. Is there independent expert evaluation of a blinded sample of generated answers? In particular, does it support the relative performance of full-data, scalar-q50, and rubric-q50 training?

  3. Which choices were made using the 394-question validation set, and which were made after examining test results? Do the main findings hold on an untouched holdout with the selection rules fixed in advance?

4 comments
AuthorsOpenReview ↗

Matched Training Controls and Held-Out Validation

We thank the reviewer for the constructive feedback and for recognizing the value of our exposure accounting and rubric diagnostics. We agree that the original binary-versus-graded comparison simultaneously changes question coverage, pair composition, and training exposure. We have therefore added new controlled experiments and clarified the limits of our conclusions in the revised manuscript.

1. Additional controls for pair density and training exposure

New Section 5.5, Table 6, and Appendix G.1 report a separate source-grouped experiment with 2,394 training questions. All DPO conditions use the same SFT initialization, backbone (Qwen3.5-9B), effective batch size (16), and nominal optimization hyperparameters.

The one-pair random condition samples one valid comparison per question and repeats it for six epochs (14,364 pair exposures; 900 updates). The six-pair condition processes 14,223 distinct valid comparisons in one epoch (889 updates).

ConditionSeed 42Seed 43786-question holdout
Random one-pair93.2193.0293.30
Full six-pair92.4892.8392.64

The six-pair condition did not outperform the one-pair control in either training seed. Its score difference relative to one-pair random was −0.73 and −0.19 on the 820-question test set.

This is an approximately exposure- and update-matched comparison, not a perfectly matched intervention. Pair repetition, distinct-pair diversity, and small update-count differences remain. Also, the new one-pair condition is not identical to the original binary condition with a score-gap threshold of 15. We explicitly distinguish these protocols in Appendix G.1.

2. Evaluation on a previously unused held-out split

To examine test-set dependence, we evaluated the existing seed-42 checkpoints on another 786 questions from 54 reference sources, disjoint from the corresponding training and test reference sources.

The checkpoints and comparison conditions were fixed before this evaluation; the additional questions were not used for training, hyperparameter tuning, or checkpoint selection.

The six-pair-minus-one-pair difference was −0.66, closely matching the −0.73 observed on the original 820-question test set. We report this as a second held-out evaluation within the same resource, not as evidence of cross-dataset generalization (Appendix G.2).

3. Rubric-selection results and exploratory decisions

We agree that Table 4 shows numerically higher three-seed means for rubric-absolute (92.78) and consensus (92.82) selection than for scalar selection (92.35).

We have not interpreted these results as evidence that rubric selection is ineffective. Rather, the limited number of training seeds does not establish a reliable advantage or a general variance ordering.

We also acknowledge the reviewer's concern about exploratory choices. Section 7 explicitly identifies the original q50 emphasis, consensus-rule interpretation, extended-training interpretation, and pair-position analyses as having been examined using the reporting test results. We do not regard these analyses as prospectively confirmed. The additional 786-question split was instead evaluated after the supplementary model conditions had been fixed.

4. Independent evaluation

We have clarified in Section 4.2 and Appendix G.5 that dataset construction included human validation of 800 original candidate answers by three materials-science evaluators, with Pearson correlation 0.822 and QWK 0.796 against the construction-stage fused scores.

We recognize that this is not independent human evaluation of the newly generated outputs. The frozen downstream evaluator provides a consistent comparison scale across checkpoints, but it may retain systematic bias. We have made this distinction explicit rather than claiming that the supplementary experiments eliminate the evaluation limitation.

We hope the additional matched-budget comparison, held-out replication, and clearer separation of exploratory versus confirmatory evidence address the reviewer's central concerns while preserving appropriately qualified conclusions.

I appreciate the authors' rebuttal. The matched-budget experiments substantially address W1/Q1. W2/Q2: The main concern is whether the evaluator can detect the benefits of richer supervision. Similar ScoreLLM scores do not necessarily imply similar scientific answer quality. The human validation covers original candidates, not the newly generated outputs. What independent evidence shows that ScoreLLM can capture improvements in scientific correctness or reasoning that richer supervision might produce? W2/Q3: The additional holdout addresses test-set dependence for the pair-density comparison, but it uses the same evaluator. It therefore does not resolve whether the observed lack of improvement reflects the supervision or the evaluation. The revised experiments strengthen the reported observations, but their meaning for scientific answer quality remains uncertain. Since this interpretation is central to the paper's contribution, I maintain my rating of 2 (Weak rejection).

We thank the reviewer for the follow-up and for acknowledging that the matched-budget experiments substantially address W1/Q1. We agree that the remaining central question is whether the evaluator can detect scientific reasoning improvements potentially introduced by richer supervision. To investigate this concern, we conducted additional cross-evaluator robustness, dimension-level, and score-resolution analyses using the existing model outputs.

1. Cross-evaluator robustness and evaluation-data separation (W2/Q2–Q3).

We trained two additional frozen evaluators alongside the original ScoreModel412 (J1), using the same Qwen3.5-4B backbone and scoring protocol but different training partitions:

  • J2: Trained on 2,394 source-grouped questions, with zero training-question overlap with both the 820-question test set and the 786-question held-out set.
  • J3: Trained on a size-matched, 2,394-question subset of J1's original training partition, controlling for scorer-training data size.

We re-evaluated the existing generations without retraining the generators. For the approximately budget-matched six-pair versus random one-pair DPO comparison, the score differences (six-pair minus one-pair) on the 820-question test set were −0.73 (J1), −0.80 (J2), and −0.67 (J3). On the additional 786-question held-out set, the corresponding differences were −0.66, −0.80, and −0.69. Thus, the observed direction persists even under J2, which has no training-question overlap with either evaluation partition.

2. Does aggregate scoring obscure improvements in scientific reasoning? (W2/Q2)

We further examined all 17 task-specific rubric dimensions using question-level paired bootstrap 95% confidence intervals (2,000 resamples) and two-sided paired permutation tests with BH-FDR correction across dimensions (q < 0.05).

For the six-pair versus random one-pair comparison, 12/17, 13/17, and 14/17 dimensions showed significant negative differences under J1, J2, and J3, respectively, with no significantly positive dimensions under any evaluator. Under J2, negative differences included scientific dimensions such as technical rationality, data support, mechanistic clarity, and failure-diagnosis accuracy.

Question-level comparisons also support this pattern: under J2, six-pair DPO outperformed random one-pair DPO on 165 questions, tied on 249, and underperformed on 406. These findings suggest that the aggregate result is not simply masking positive effects across the measured scientific rubric dimensions.

3. Evaluator sensitivity and remaining limitations.

We additionally examined score saturation. Exact maximum scores were extremely rare, indicating no substantial hard-ceiling effect, although approximately 30% of six-pair/random-one-pair comparisons received identical total scores. As internal positive controls, all three evaluators consistently detected larger improvements from optimized versus raw training data (approximately +3.4 to +4.2 points).

Importantly, we do not claim that these analyses establish independent scientific correctness or statistical equivalence between training methods. The three evaluators share a model family and annotation targets, and independent expert assessment of newly generated answers remains unavailable. Nevertheless, the new analyses directly strengthen the evidence that the observed lack of a six-pair advantage is robust to scorer-training partitions, is not driven by obvious score saturation, and is not solely an artifact of aggregating the evaluated scientific dimensions.

We appreciate the reviewer's emphasis on distinguishing evaluation robustness from independent scientific validity, and have explicitly maintained this distinction in interpreting our findings.

Thank you for the additional analyses. They strengthen the six-pair versus one-pair comparison.

W2/Q2: The scorers agree on the direction of the effect, but they share the same annotation targets. It remains unclear whether the observed score differences reflect differences in scientific correctness or reasoning.

Q3: The new holdout checks the effect of pair density. It does not show whether rubric dimensions add useful information beyond the scalar score when the two selection rules disagree.

The new analyses strengthen the pair-density result, but the evidence remains insufficient to determine whether the score differences reflect scientific answer quality and whether multidimensional rubrics add value beyond scalar scores. I therefore maintain my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.