acceptodds
Withdrawn from ICLR 2027

HSIR: Hidden-State Intrinsic Rewarding for Multilingual Alignment

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models (LLMs) exhibit strong yet uneven multilingual capabilities, motivating alignment methods that improve cross-lingual consistency without costly external evaluators. Existing approaches often derive rewards from translation models, multilingual encoders, or stronger language models, incurring additional inference costs and introducing a mismatch between external reward spaces and the policy model. We propose HSIR, a Hidden-State Intrinsic Rewarding framework for multilingual alignment. HSIR constructs an intrinsic reward directly from the policy model's hidden representations by measuring the similarity between generated responses and reference answers, and incorporates it into Group Relative Policy Optimization (GRPO). This enables multilingual alignment without an additional reward model or translation evaluator. Experiments on multilingual open-ended generation show that HSIR consistently outperforms supervised fine-tuning and alternative semantic, cross-lingual, and translation-based rewards. On Qwen2.5-3B-Instruct and Llama-3.2-3B-Instruct, HSIR achieves All-avg win rates of 49.02% and 66.32%, respectively, corresponding to improvements of 21.58 and 20.31 percentage points over the original models. These results demonstrate that hidden representations provide an effective intrinsic reward for efficient multilingual alignment.

Primary area:
reinforcement learning
Keywords:
Multilingual Alignment, Large Language Models, Reinforcement Learning, Intrinsic Reward

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_HcviOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether a language model's own representations can replace a separately trained reward model or an external multilingual encoder during multilingual post-training. HSIR prepares target-language references by translating English answers offline. During training, a frozen copy of the initial policy encodes both a generated continuation and its corresponding reference under the same prompt. The method mean-pools continuation states at each selected layer, computes a cosine similarity per layer, averages these scores, and uses the resulting reward in otherwise standard GRPO. The contribution is therefore a reference-conditioned reward construction and its application to multilingual alignment, rather than a new policy optimizer or a supervision-free alignment method.

The experiments train Qwen2.5-3B-Instruct and Llama-3.2-3B-Instruct on a 1.5K-example alignment set spanning five languages. OMGEval evaluates instruction following in those languages and in Bengali and German; MGSM8K evaluates transfer to multilingual mathematics. HSIR reports the highest OMGEval All-avg scores, 49.02% and 66.32%, improving on the strongest reported baseline by 0.50 and 4.46 percentage points, respectively. Mathematical transfer is mixed: HSIR is best on average for Qwen but trails several reward baselines for Llama. Further experiments examine layer selection, aggregation, response-group size, data scale, and a logit-based reward.

Critical strengths
  1. A simple, well-defined reward with a useful implementation motivation. Sections 3.1–3.3 specify a frozen initial-policy scorer, continuation-only pooling, per-layer cosine normalization, and integration with standard GRPO. This avoids requiring a separately trained reward head or a dedicated online translation/embedding model. The fixed references make the supervision source explicit, and the design is straightforward to implement. The anonymous code/data link is also a positive reproducibility commitment, although I have not inspected it.
  2. Promising comparative evidence, especially on Llama, with relevant controlled variants. Table 1 shows a 4.46-point Llama improvement over the best reported baseline, alongside a smaller Qwen improvement. Section 4.2 includes translation- and embedding-based reward variants optimized with GRPO, which helps distinguish reward construction from optimizer choice. Sections 5.2–5.7 examine several design choices rather than reporting only the best configuration. The discussion also explicitly acknowledges that mathematical transfer is model-dependent, avoiding a claim of universal improvement across tasks.
Critical weaknesses
  1. The main evaluation is insufficiently specified and lacks uncertainty estimates, so reliable superiority is not established. Section 4.1 identifies GPT-3.5-turbo as the comparison opponent and mentions an extension of AlpacaEval 2.0, but does not identify the actual evaluation judge/version, judging prompt, position handling, tie handling, whether win rates are length-controlled, test decoding settings, or evaluated sample counts. Neither Table 1 nor Table 2 reports confidence intervals or variation across training seeds. This is consequential because the headline Qwen gain is only 0.50 points on OMGEval and 0.28 points over the best mathematical baseline. The larger Llama result is encouraging, but its reliability also cannot be assessed from the reported information. The paper should provide the complete multilingual evaluation configuration and paired uncertainty estimates, and repeat the key comparisons across seeds. It should also define and justify All-avg's equal weighting of the ID and OOD groups and report the ordinary per-language macro-average.
  2. The central semantic/cross-lingual interpretation of the reward is not independently validated. Equations 2–4 compare a response with one reference in the same language; they do not directly compare responses across languages. Tables 1–2 measure judged answer quality and mathematical accuracy, not consistency for equivalent questions across languages. The analysis does not establish that hidden-state cosine similarity prefers correct answers over fluent but incorrect reference-like text, or tolerates valid answers expressed differently. Section 5.1's separately min–max-normalized training-reward curves show optimization of the respective proxies; a higher normalized plateau cannot establish a more informative semantic signal across reward types. Likewise, Table 5 shows a closer paraphrase of the reference, but does not establish that the original model's additional statement is incorrect. This limits the support for the proposed explanation about overcoming external reward-space mismatch. Independent reward–quality agreement tests, controlled perturbations such as negation and factual errors, valid alternative answers, and a direct cross-lingual consistency evaluation would substantially strengthen the paper. At minimum, the claims should be narrowed to competitive multilingual reference-based optimization. The Qwen Bengali regression from 29.94% to 21.49% in Table 1 should also be analyzed when discussing transfer to unseen languages.
Questions for authors
  1. What exact judge, model version, multilingual prompts, ordering procedure, tie handling, length-control setting, sample counts, and generation settings produced Table 1? Do the 0.50-point Qwen and 4.46-point Llama improvements remain reliable under paired confidence intervals and repeated training runs?
  2. Does HSIR correlate with independently assessed correctness/semantic quality when candidate answers include valid paraphrases, valid alternative answers, negations, and plausible factual errors? Is there direct evidence of improved consistency across equivalent prompts in different languages, rather than only improved judged quality within each language?
  3. How were the 1.5K training examples and English references selected, how were target-language references translated and checked, and what is the per-language allocation? Were the same applicable reference answers, examples, rollout budgets, and tuning budgets used for the controlled GRPO baselines? These details are not clearly specified and affect whether the gain can be attributed to the reward representation.
Reviewer_wvtiOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

Paper focus on the problem of mismatch between external reward space and policy model representation space. Paper introduces HSIR, which is supervised hidden state rewarding framework using an pivot language (english in the paper). HSIR uses this reward to align policy with expected reward and trains model on GRPO and evalutes 2 sets of models on multiple langauges and tasks

Critical strengths
  1. The idea is simple and intuitive, paper has extensive experiments and results are positive over multiple baselines.
Critical weaknesses
  1. My main concern is that the proposed approach appears engineered, while its conceptual novelty seems very limited.
Questions for authors
  1. Can you provide an study(theoritical and experimental examples if possible) how does external rewards if adapted for multilinguality effects GRPO training, and how does HSIR still overcome those problems?, give me pitfall of different external rewards and why hidden state cosine similarty is better here
  2. How much effect the cross-linguality have in the gains, is there a way to visualize or explain that?
  3. In the figure 2, the MAPO has weakest possible reward vs steps trend, but still it is the 2nd best performing approch in Table 1, is there an explanation for that?
  4. what is the the overall compute requirement for HSIR, compared to other methods?
  5. are all the layers necessary while calculating rewards, what happens when i switch off few layers, while reward calculation?
Reviewer_eLTCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HSIR, a reference-based reward that measures the similarity between the hidden representations of generated and reference responses, averaging cosine similarities across layers of a frozen copy of the initial policy. Combined with GRPO, it improves multilingual instruction following on two 3B models, with stronger gains on Llama and mixed transfer to mathematical reasoning.

Critical strengths
  1. The paper proposes a simple reward design without a separately trained reward head or online translation evaluator.
  2. The experiments demonstrate clear gains on Llama’s multilingual instruction following, including languages unseen during alignment.
  3. The experiments provide useful ablations on layer selection, aggregation, sampling count, data scale, and logit-based rewards.
Critical weaknesses
  1. The main change from reference-similarity rewards is the scoring representation, and the specific benefit of using the initial policy’s own hidden space is not isolated.
  2. The paper does not directly test whether pooled hidden-state similarity tracks correctness and helpfulness rather than wording, topic, or style.
  3. Qwen’s gains over the strongest baselines are small, i.e., 0.50 points on OMGEval and 0.28 points on multilingual math.
  4. The normalized reward curves show that HSIR’s score increases during training, but do not show whether it tracks response quality more reliably than other rewards.
Questions for authors
  1. How well does HSIR rank independently judged response quality, especially for answers with small factual errors, negation, or missing key information?
  2. Which models produce the translated references and judge OMGEval, and do the gains hold with length-controlled evaluation?
  3. Do the small Qwen gains hold across training seeds and paired confidence intervals?
  4. What are the training time and peak memory costs compared with CM-Alignmulti, including the frozen scoring model?
  5. With references and GRPO fixed, does the initial policy outperform a similarly sized alternative scoring model, and how sensitive is HSIR to reference quality?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.