- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces ChiEngMixBench, a collection of 1,167 closely matched sentence pairs containing Chinese or English realizations of technical terminology. It compares four likelihood-based scoring definitions, combining whole-sentence versus target-span scoring with summed versus mean token log-probabilities. Across four models, aggregate preference directions remain relatively stable, while individual preference decisions can differ substantially. For Qwen2.5-7B, the largest disagreement reaches 19.83%.
Critical strengths
The controlled comparison isolates two concrete scoring choices while using the same underlying token probabilities. This makes the measurement issue clear and the results straightforward to interpret. The item-level analysis adds useful information beyond aggregate preference rates. For Qwen2.5-7B, the English-preferred sets under whole-sentence sum and target-span mean have a Jaccard similarity of only 0.083, despite agreeing on the majority preference direction.
Critical weaknesses
The empirical analysis mainly establishes that scoring definitions disagree, with limited explanation of the observed flips. A clearer account of the roles of token counts, target-span scores, and surrounding-context scores would strengthen the paper’s substantive contribution. The described release omits the source sentences and term-normalization records. Although metadata and numerical outputs support inspection of the reported comparisons, the procedure for obtaining or reconstructing the sentence pairs is unclear. This limits the benchmark’s usefulness for subsequent model evaluation.
Questions for authors
Using the existing outputs, could you explain representative preference flips by separating the effects of token count, target-span likelihood, and scores outside the target span? How can researchers obtain or reconstruct the underlying sentence pairs? Please specify the available access procedure or reconstruction materials.
Thank you for the detailed response. The token-level examples clarify how token counts, aggregation, and contextual contributions lead to preference flips, while the boundary sensitivity analysis shows that item-level disagreements persist under the revised span definition. The clarified access to the frozen sentence pairs also addresses my concern about the benchmark’s availability for further evaluation. These additions substantially address my original concerns and strengthen the paper’s empirical support. I am therefore raising my rating to Weak Accept.