- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies whether a language model's own representations can replace a separately trained reward model or an external multilingual encoder during multilingual post-training. HSIR prepares target-language references by translating English answers offline. During training, a frozen copy of the initial policy encodes both a generated continuation and its corresponding reference under the same prompt. The method mean-pools continuation states at each selected layer, computes a cosine similarity per layer, averages these scores, and uses the resulting reward in otherwise standard GRPO. The contribution is therefore a reference-conditioned reward construction and its application to multilingual alignment, rather than a new policy optimizer or a supervision-free alignment method.
The experiments train Qwen2.5-3B-Instruct and Llama-3.2-3B-Instruct on a 1.5K-example alignment set spanning five languages. OMGEval evaluates instruction following in those languages and in Bengali and German; MGSM8K evaluates transfer to multilingual mathematics. HSIR reports the highest OMGEval All-avg scores, 49.02% and 66.32%, improving on the strongest reported baseline by 0.50 and 4.46 percentage points, respectively. Mathematical transfer is mixed: HSIR is best on average for Qwen but trails several reward baselines for Llama. Further experiments examine layer selection, aggregation, response-group size, data scale, and a logit-based reward.
Critical strengths
- A simple, well-defined reward with a useful implementation motivation. Sections 3.1–3.3 specify a frozen initial-policy scorer, continuation-only pooling, per-layer cosine normalization, and integration with standard GRPO. This avoids requiring a separately trained reward head or a dedicated online translation/embedding model. The fixed references make the supervision source explicit, and the design is straightforward to implement. The anonymous code/data link is also a positive reproducibility commitment, although I have not inspected it.
- Promising comparative evidence, especially on Llama, with relevant controlled variants. Table 1 shows a 4.46-point Llama improvement over the best reported baseline, alongside a smaller Qwen improvement. Section 4.2 includes translation- and embedding-based reward variants optimized with GRPO, which helps distinguish reward construction from optimizer choice. Sections 5.2–5.7 examine several design choices rather than reporting only the best configuration. The discussion also explicitly acknowledges that mathematical transfer is model-dependent, avoiding a claim of universal improvement across tasks.
Critical weaknesses
- The main evaluation is insufficiently specified and lacks uncertainty estimates, so reliable superiority is not established. Section 4.1 identifies GPT-3.5-turbo as the comparison opponent and mentions an extension of AlpacaEval 2.0, but does not identify the actual evaluation judge/version, judging prompt, position handling, tie handling, whether win rates are length-controlled, test decoding settings, or evaluated sample counts. Neither Table 1 nor Table 2 reports confidence intervals or variation across training seeds. This is consequential because the headline Qwen gain is only 0.50 points on OMGEval and 0.28 points over the best mathematical baseline. The larger Llama result is encouraging, but its reliability also cannot be assessed from the reported information. The paper should provide the complete multilingual evaluation configuration and paired uncertainty estimates, and repeat the key comparisons across seeds. It should also define and justify All-avg's equal weighting of the ID and OOD groups and report the ordinary per-language macro-average.
- The central semantic/cross-lingual interpretation of the reward is not independently validated. Equations 2–4 compare a response with one reference in the same language; they do not directly compare responses across languages. Tables 1–2 measure judged answer quality and mathematical accuracy, not consistency for equivalent questions across languages. The analysis does not establish that hidden-state cosine similarity prefers correct answers over fluent but incorrect reference-like text, or tolerates valid answers expressed differently. Section 5.1's separately min–max-normalized training-reward curves show optimization of the respective proxies; a higher normalized plateau cannot establish a more informative semantic signal across reward types. Likewise, Table 5 shows a closer paraphrase of the reference, but does not establish that the original model's additional statement is incorrect. This limits the support for the proposed explanation about overcoming external reward-space mismatch. Independent reward–quality agreement tests, controlled perturbations such as negation and factual errors, valid alternative answers, and a direct cross-lingual consistency evaluation would substantially strengthen the paper. At minimum, the claims should be narrowed to competitive multilingual reference-based optimization. The Qwen Bengali regression from 29.94% to 21.49% in Table 1 should also be analyzed when discussing transfer to unseen languages.
Questions for authors
- What exact judge, model version, multilingual prompts, ordering procedure, tie handling, length-control setting, sample counts, and generation settings produced Table 1? Do the 0.50-point Qwen and 4.46-point Llama improvements remain reliable under paired confidence intervals and repeated training runs?
- Does HSIR correlate with independently assessed correctness/semantic quality when candidate answers include valid paraphrases, valid alternative answers, negations, and plausible factual errors? Is there direct evidence of improved consistency across equivalent prompts in different languages, rather than only improved judged quality within each language?
- How were the 1.5K training examples and English references selected, how were target-language references translated and checked, and what is the per-language allocation? Were the same applicable reference answers, examples, rollout budgets, and tuning budgets used for the controlled GRPO baselines? These details are not clearly specified and affect whether the gain can be attributed to the reward representation.