acceptodds
Under review as a conference paper at ICLR 2027

Cross-Lingual Prompt Steerability: Towards Accurate and Robust LLM Behavior across Languages

Abstract

System prompts provide a lightweight yet powerful mechanism for conditioning large language models (LLMs) at inference time. While prior work has focused on English‑only settings, real‑world deployments benefit from a single prompt that operates reliably across languages. This paper presents a comprehensive study of how different system prompts steer models toward accurate and robust cross-lingual behavior. We propose a unified four-dimensional evaluation framework to assess system prompts in multilingual environments. Through experiments on five languages, three LLMs, and three benchmarks, we uncover that certain prompt components, such as CoT, emotion, and scenario, correlate with robust multilingual behavior. We develop a prompt optimization framework for multilingual settings and show it can automatically discover prompts that improve all metrics by 5-10%. Finally, we analyze over 10 million reasoning units and find that more performant system prompts induce more structured and consistent reasoning patterns, while reducing unnecessary language-switching. Together, we highlight system prompt optimization as a scalable path to accurate and robust multilingual LLM behavior.

Primary area:
foundation or frontier models, including LLMs
Keywords:
prompt tuning, code-switching, multilinguality, multilingual evaluation, cross-lingual transfer

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gUhfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether a single system prompt can improve accuracy and behavioral stability across languages and tasks. It evaluates each prompt using four metrics: Acc_mean, Acc_var, Consistency, and Len_var; it min–max normalizes them, weights them by 0.5, 0.25, 0.125, and 0.125, and reverses the two variance terms. The authors build 10,000 prompt components from ten categories, randomly compose 1,000 prompts, evaluate them, and train an XLM-RoBERTa reward model to predict a four-dimensional reward vector with a max-margin pairwise loss. They then adapt the component-editing and iterative-selection procedure from SPRIG [1] for a 25-step optimization of English system prompts; the downstream LLM is not updated, and the selected prompt is prepended at inference time. The main experiments use Gemma-3-12B-Instruct, Llama-3.1-8B-Instruct, and Qwen2.5-7B-Instruct on MMLU-Pro, MATH500, and UniMoral, with additional transfer to MGSM and analyses of segmented reasoning units using language identification and reasoning-type classifiers.

[1] Lechen Zhang et al. SPRIG: Improving Large Language Model Performance by System Prompt Optimization. arXiv:2410.14826, 2024.

Critical strengths
  1. The evaluation has substantial coverage with a clean item-level split. The main study spans three LLMs, three benchmarks, and five languages. It assigns all language versions of a question to the same split, uses training questions for reward-model labels, and computes optimized-prompt results on held-out test questions. This prevents language variants of the same question from crossing partitions and makes the end-to-end comparison more informative.

  2. The extension experiments test transfer beyond the main search setting. The MGSM experiment reuses prompts optimized on the main tasks and evaluates them on an unseen benchmark and seven unseen languages; on Qwen2.5-7B, Acc_mean increases from 0.636 to 0.674, although Acc_var changes little.

Critical weaknesses
  1. The paper does not establish that the four-metric objective adds value beyond accuracy-only SPRIG [1]. Len_var is affected by tokenization and language-specific expressions, while Consistency counts identical wrong answers as consistent. Without an accuracy-only SPRIG run under the same search and evaluation budget, the gains cannot be attributed to the added metrics. Table 9 also contains counterexamples to the claim that all metrics improve by 5–10%: accuracy and other metrics trade off on Llama–MATH500, and Gemma–UniMoral accuracy decreases. A narrower aggregate claim is needed.

  2. The reasoning-behavior analysis does not establish a mechanism for the gains. It follows the segmentation and classification workflow in Language Matters [2]; the 70B relabeling check gives only κ = 0.528 and does not calibrate errors by language. The higher Hindi Others rate may reflect classifier or segmentation bias, and the PCA/regression results are confounded by output length and prompt style. The evidence, therefore, supports correlation, not the claim that reasoning patterns drive robustness; language-wise annotation checks and confound controls are needed.

[2] Zhi Rui Tam et al. Language Matters: How Do Multilingual Input and Reasoning Paths Affect Large Reasoning Models? arXiv:2505.17407, 2025.

Questions for authors
  1. Under the same component library, reward-model training data, candidate-evaluation budget, and search rounds, how does the four-metric objective compare with an accuracy-only SPRIG run? Can you report all four raw metrics and the accuracy of the weakest language? This comparison would directly determine the incremental value of the proposed objective.

  2. Maybe you can manually audit segmentation and category labels on samples from the five main languages, especially the distinction between Others, Logical Reasoning, and Reframing? After relabeling the key language comparisons with the 70B model and controlling for total length, task, and model, do the main conclusions remain?

4 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

We sincerely thank the reviewer for the thoughtful and constructive feedback. Below, we respond to each point and include several additional results that we hope will clarify your concerns.


W1. The paper does not establish that the four-metric objective adds value beyond accuracy-only SPRIG [1]. Len_var is affected by tokenization and language-specific expressions, while Consistency counts identical wrong answers as consistent. Without an accuracy-only SPRIG run under the same search and evaluation budget, the gains cannot be attributed to the added metrics. Table 9 also contains counterexamples to the claim that all metrics improve by 5–10%: accuracy and other metrics trade off on Llama–MATH500, and Gemma–UniMoral accuracy decreases. A narrower aggregate claim is needed.

Q1. Under the same component library, reward-model training data, candidate-evaluation budget, and search rounds, how does the four-metric objective compare with an accuracy-only SPRIG run? Can you report all four raw metrics and the accuracy of the weakest language? This comparison would directly determine the incremental value of the proposed objective.

Thank you for this valuable suggestion! We agree that our work builds upon SPRIG as the underlying optimizer. However, our main contribution is extending system prompt optimization to the multilingual, multi-objective setting, where a single prompt must achieve both high accuracy and robust behavior across languages. Beyond optimization, we also introduce a four-dimensional evaluation framework and provide systematic analyses of multilingual prompt components and reasoning behaviors, neither of which was explored in the original SPRIG.

To better isolate these contributions, we conducted additional experiments comparing (1) English-only accuracy optimization, (2) multilingual accuracy optimization, and (3) our four-metric optimization, using the same SPRIG optimizer and English prompt search space. For a fair comparison, we re-evaluated all methods on the same held-out questions under the same computational environment, so some numerical results may differ slightly from those reported in the original manuscript. The results are summarized below.

Model / MetricEnglish-onlyMultilingual accuracyOurs (4 metrics)
Gemma-3-12B
Accuracy ↑0.6960.7000.703
Accuracy variance ↓0.00650.00650.0061
Consistency ↑0.4980.5000.511
Length variance ↓87K124K117K
English accuracy ↑0.7420.7400.740
Other 4 languages ↑0.6860.6880.693
Llama-3.1-8B
Accuracy ↑0.4160.4100.412
Accuracy variance ↓0.01380.01440.0141
Consistency ↑0.2140.2120.214
Length variance ↓718K730K707K
English accuracy ↑0.5500.5340.536
Other 4 languages ↑0.3820.3790.382

These results provide several important observations. First, our four-metric optimization outperforms both accuracy-only objectives on Gemma, improving overall accuracy (0.703 vs. 0.700), accuracy variance (0.0061 vs. 0.0065), and consistency (0.511 vs. 0.50). The accuracy improvement is primarily driven by non-English languages, while English performance remains comparable. Second, on Llama, the three approaches achieve broadly similar results, with our method attaining the lowest output-length variance. These results suggest that multilingual, multi-objective optimization can provide improvements beyond accuracy-only SPRIG.

Regarding the metric concerns, we agree that consistency can count identical incorrect answers. However, this is intentional, as consistency measures agreement across languages independently of correctness, which is already captured by accuracy (receiving the highest weight of 0.5). Similarly, while tokenization efficiency varies across languages, our length metric measures computational reasoning effort, as each generated token incurs approximately comparable inference cost within the same model. We will clarify these distinctions in the revision.

Finally, we acknowledge that improvements are not uniform across all models and benchmarks, as illustrated by the Llama–MATH500 and Gemma–UniMoral results in Table 9. We will clarify that the reported 5–10% improvements describe aggregate trends, while individual settings may exhibit trade-offs across metrics.

We will incorporate these ablation results and clarifications into the revised manuscript.

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

W2. The reasoning-behavior analysis does not establish a mechanism for the gains. It follows the segmentation and classification workflow in Language Matters [2]; the 70B relabeling check gives only κ = 0.528 and does not calibrate errors by language. The higher Hindi Others rate may reflect classifier or segmentation bias, and the PCA/regression results are confounded by output length and prompt style. The evidence, therefore, supports correlation, not the claim that reasoning patterns drive robustness; language-wise annotation checks and confound controls are needed.

Q2. Maybe you can manually audit segmentation and category labels on samples from the five main languages, especially the distinction between Others, Logical Reasoning, and Reframing? After relabeling the key language comparisons with the 70B model and controlling for total length, task, and model, do the main conclusions remain?

Thank you for these suggestions! We would like to clarify that our reasoning analysis aims to characterize observable changes in reasoning patterns following prompt optimization, without claiming to establish a causal mechanism for performance improvements.

Regarding classification reliability, we had already manually inspected reasoning-category labels across all five languages before conducting our analysis. In particular, we noticed the relatively high proportion of Others in Hindi and specifically examined these cases, finding no substantial systematic categorization errors. We will make this validation process clearer in the revision.

To further strengthen this validation, we conducted an independent, blinded annotation audit using Claude Opus 5.5, covering 100 responses (705 reasoning units, 20 responses per language). The model assigned reasoning categories without access to the original Qwen3-14B or LLaMA-3.1-70B-Instruct labels. The following table reports step-level Cohen's κ:

ComparisonAllENESFRHIZH
Qwen vs. LLaMA (15K sample)0.550.580.580.550.530.29
Qwen vs. Opus0.470.600.540.550.420.24
LLaMA vs. Opus0.600.710.500.510.570.69

Although agreement varies across languages, particularly for Qwen's Chinese annotations, the independent audit supports our observation regarding Hindi. 96% of the Hindi units classified as Others by Qwen were also identified as such in the audit. Opus assigned 43% of Hindi units to Others, compared with 29% for English, suggesting that the elevated Hindi Others rate is not simply a classification artifact.

Regarding the robustness of our findings, we re-examined the key reasoning comparisons using LLaMA-3.1-70B-Instruct annotations on the 15K sample, including analyses controlling for response length and task × language. Our central findings remain largely consistent:

  • Others and accuracy: Others is negatively associated with accuracy under both classifiers, and this relationship remains significant after controlling for response length and task × language (p < 0.001).
  • Language-specific patterns: English exhibits the highest Reframing frequency under both classifiers. Hindi also has the highest Others proportion when grouped by task language, although the difference is smaller under LLaMA (31.3% versus 30.9% for French).
  • Task-dependent reasoning: Calculation remains positively and significantly associated with accuracy on MATH500 under both classifiers.

Some finer-grained patterns are less stable. For example, the negative association between Reframing and accuracy remains significant at the prompt level under both classifiers, but loses significance under LLaMA after controlling for length and task. We will clarify these nuances and the language-dependent annotation reliability in the revision.

Overall, the central associations between reasoning behaviors and task performance remain robust across classifiers and statistical controls, supporting the descriptive conclusions of our reasoning analysis.

AuthorsOpenReview ↗

Follow-up on our rebuttal

Dear Reviewer guHf,

Thank you again for your valuable feedback! We hope our responses and additional experiments have adequately addressed your questions. We would be happy to address any further questions or suggestions you may have. If you feel your concerns have been resolved, we would greatly appreciate your consideration in updating your assessment. Thank you again for your time and thoughtful review!

AuthorsOpenReview ↗

Second Follow-up on the rebuttal

Dear Reviewer gUhf,

As the discussion period is approaching its end, we wanted to check whether you have had a chance to review our rebuttal and whether any concerns remain. We would be happy to provide further clarification if needed, and would appreciate your consideration of an updated rating if our responses have addressed your concerns. Thank you!

Reviewer_EfWyOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how system prompts affect multilingual LLM behavior. It evaluates 1,000 randomly composed system prompts on 3 models, 3 benchmarks, and 5 languages using average accuracy, accuracy variance, cross-lingual consistency, and output length variance. It adapts SPRIG with a multi-dimensional reward model to optimize English prompts against multilingual performance objectives. The paper also examines generalization beyond the main setting and analyzes language use and reasoning-unit distributions before and after optimization.

Critical strengths
  1. The paper addresses a practical problem and systematically evaluates system prompts across multiple languages, model families, and complementary performance metrics.
  2. The held-out question evaluation and supplementary experiments on an unseen benchmark, additional languages, lower-resource settings, and a larger model provide useful evidence beyond the main experimental setting.
Critical weaknesses
  1. The incremental contribution over SPRIG is insufficiently isolated. The method largely follows the existing optimizer, while the comparisons with standard prompts and best-random baselines do not establish how much benefit comes specifically from multilingual optimization or the four-metric objective.
  2. The reasoning analysis remains primarily descriptive. Changes in reasoning-unit counts and the Other category may reflect shorter outputs or the removal of redundant instructions rather than improved reasoning. These alternatives are not adequately controlled.
  3. The generality of the component-level and reasoning-behavior findings is limited. The main analyses still rely on earlier-generation instruction-tuned models. The additional optimization result does not establish whether the component effects and reasoning patterns persist in newer models, particularly reasoning models.
Questions for authors
  1. Could you compare English-only accuracy optimization, multilingual accuracy optimization, and the proposed four-metric optimization under the same English prompt search space and evaluation budget? This would distinguish the benefit of multilingual supervision from that of the composite objective and the underlying SPRIG optimizer.
  2. How was the approximately 0.8 correlation between repeated evaluations estimated? Were the same questions evaluated repeatedly, or were different question samples or bootstrap samples used?
4 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

We sincerely thank the reviewer for the thoughtful and constructive feedback. Below, we respond to each point and include several additional results that we hope will clarify your concerns.


W1. The incremental contribution over SPRIG is insufficiently isolated. The method largely follows the existing optimizer, while the comparisons with standard prompts and best-random baselines do not establish how much benefit comes specifically from multilingual optimization or the four-metric objective.

Q1. Could you compare English-only accuracy optimization, multilingual accuracy optimization, and the proposed four-metric optimization under the same English prompt search space and evaluation budget? This would distinguish the benefit of multilingual supervision from that of the composite objective and the underlying SPRIG optimizer.

Thank you for this valuable suggestion! We agree that our work builds upon SPRIG as the underlying optimizer. However, our main contribution is extending system prompt optimization to the multilingual, multi-objective setting, where a single prompt must achieve both high accuracy and robust behavior across languages. Beyond optimization, we also introduce a four-dimensional evaluation framework and provide systematic analyses of multilingual prompt components and reasoning behaviors, neither of which was explored in the original SPRIG.

To better isolate these contributions, we conducted additional experiments comparing (1) English-only accuracy optimization, (2) multilingual accuracy optimization, and (3) our four-metric optimization, using the same SPRIG optimizer and English prompt search space. For a fair comparison, we re-evaluated all methods on the same held-out questions under the same computational environment, so some numerical results may differ slightly from those reported in the original manuscript. The results are summarized below.

Model / MetricEnglish-onlyMultilingual accuracyOurs (4 metrics)
Gemma-3-12B
Accuracy ↑0.6960.7000.703
Accuracy variance ↓0.00650.00650.0061
Consistency ↑0.4980.5000.511
Length variance ↓87K124K117K
English accuracy ↑0.7420.7400.740
Other 4 languages ↑0.6860.6880.693
Llama-3.1-8B
Accuracy ↑0.4160.4100.412
Accuracy variance ↓0.01380.01440.0141
Consistency ↑0.2140.2120.214
Length variance ↓718K730K707K
English accuracy ↑0.5500.5340.536
Other 4 languages ↑0.3820.3790.382

These results provide several important observations. First, our four-metric optimization outperforms both accuracy-only objectives on Gemma, improving overall accuracy (0.703 vs. 0.700), accuracy variance (0.0061 vs. 0.0065), and consistency (0.511 vs. 0.50). The accuracy improvement is primarily driven by non-English languages, while English performance remains comparable. Second, on Llama, the three approaches achieve broadly similar results, with our method attaining the lowest output-length variance. These results suggest that multilingual, multi-objective optimization can provide improvements beyond accuracy-only SPRIG. We will incorporate these ablation results into the revised manuscript.


W2. The reasoning analysis remains primarily descriptive. Changes in reasoning-unit counts and the Other category may reflect shorter outputs or the removal of redundant instructions rather than improved reasoning. These alternatives are not adequately controlled.

We would like to clarify that our reasoning analysis is intended to characterize the changes in observable reasoning patterns associated with prompt optimization, rather than establish a causal mechanism explaining performance improvements.

We agree that shorter outputs and the removal of redundant instructions may contribute to the observed changes. However, we view these as meaningful outcomes of prompt optimization rather than competing explanations. In particular, reducing unnecessary or redundant reasoning while preserving or improving task performance can itself reflect more efficient and interpretable reasoning behavior.

Moreover, our findings are not simply explained by uniformly shorter outputs. For example, on MATH500, the absolute frequencies of Calculation and Logical Reasoning both increase after optimization, despite an overall reduction in reasoning units (Appendix Table 14). This suggests that prompt optimization changes not only the quantity but also the composition of observable reasoning steps, with effects that depend on the task.

We will clarify this interpretation in the revision and emphasize the descriptive nature of our reasoning analysis.

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

W3. The generality of the component-level and reasoning-behavior findings is limited. The main analyses still rely on earlier-generation instruction-tuned models. The additional optimization result does not establish whether the component effects and reasoning patterns persist in newer models, particularly reasoning models.

Thank you for raising this important concern! To investigate whether our findings generalize to newer reasoning models, we conducted additional experiments on Qwen3.5-4B in thinking mode, evaluating 450K generations across five languages and three benchmarks, and comparing them with the three original models on the same prompts.

Our results suggest that the main component-level findings largely generalize to reasoning models, while revealing several interesting differences:

  1. System prompts remain effective. Accuracy varies substantially across prompts (0.624–0.775), and cross-lingual consistency exhibits variability comparable to earlier models (SD = 0.086). Prompt performance also transfers reasonably well, with Spearman correlations of 0.45–0.64 between Qwen3.5 and the original models.

  2. Key component effects remain consistent. Cross-language instructions (t = −11.4), style (−7.8), and behavioral components (−6.8) remain strongly detrimental to accuracy. Across statistically significant component effects in the original models, 13 of 15 comparisons agree in direction with Qwen3.5.

  3. Some effects differ for reasoning models. Notably, explicit CoT instructions slightly reduce accuracy (t = −2.1) and consistency (t = −2.6), while increasing thinking length. We believe this is expected, since models with internal reasoning capabilities may benefit less from additional explicit CoT instructions.

Regarding reasoning behavior, we also observe that higher-performing prompts tend to generate shorter thinking traces (correlation = −0.56), suggesting that prompt quality remains associated with reasoning efficiency. However, unlike earlier models, Qwen3.5 generates approximately 95–97% of its thinking in English regardless of the question language, suggesting that language-switching behaviors differ substantially in reasoning models. These new analyses use lightweight reasoning proxies and do not constitute a full replication of our reasoning-unit classification analysis.

We will incorporate these additional findings into the revision, highlighting both the generalizability of the dominant component effects and the differences introduced by reasoning models.


Q2. How was the approximately 0.8 correlation between repeated evaluations estimated? Were the same questions evaluated repeatedly, or were different question samples or bootstrap samples used?

Thank you for requesting clarification! The approximately 0.8 correlation estimates the reliability of prompt rankings under different bootstrapped question samples. Specifically, we evaluate the same set of system prompts using two independently bootstrapped samples of benchmark questions and compute the Spearman correlation between their resulting prompt performance rankings. This measures how consistently the same prompts are ranked across different evaluations, providing an empirical reference for the reliability of our reward model's predictions. We will clarify this procedure in the revised manuscript.

Thank you for the detailed response and for running the additional experiments. They helped clarify two points from my review:

  1. The three-way comparison directly addresses my question about what the 4 metric objective adds. Compared with optimizing multilingual accuracy alone, the 4 metric objective improved cross-lingual consistency on Gemma, while the gains in mean accuracy were modest on both models.
  2. The additional analysis on Qwen3.5 4B thinking model is also interesting. Many component effects were consistent with the original experiments, and some are not (CoT). This helps clarify which findings carry over to this newer reasoning model and which may depend on the model setting.

Thank you so much for your positive feedback! We're glad that the additional experiments helped clarify your concerns. Given these new results, we would appreciate it if you could reconsider your overall rating. Please let us know if you have any remaining concerns, and we'd be happy to address them!

Reviewer_XwXDOpenReview ↗
Rating
2
Confidence
1(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether system prompts can improve LLM performance across languages. The authors propose four evaluation metrics: mean accuracy across languages, accuracy variance, answer consistency, and output-length variance. They analyze how different prompt components relate to these metrics and adapt the SPRIG prompt-optimization framework for multi-objective optimization. The experiments cover three instruction-tuned models, five primary languages, and three benchmarks. The authors report that optimized prompts improve mean accuracy and cross-lingual answer consistency while reducing output-length variance. On the unseen MGSM benchmark, Qwen2.5-7B’s mean accuracy increases from 0.636 to 0.674, while accuracy variance changes very little. The paper also analyzes language use and reasoning types in model outputs, but these findings are primarily correlational and do not by themselves establish that particular reasoning processes cause the performance gains.

Critical strengths
  1. The paper addresses a relevant problem with broad empirical coverage. It systematically examines prompt language, prompt components, and multilingual performance across multiple models, languages, and task types, offering useful empirical guidance for multilingual LLM prompting.

  2. The optimization results have some evidence of generalization. The authors split data by question, keeping versions of the same question in different languages within the same split. The optimized prompts also improve accuracy on MGSM, which was not used during optimization. This provides evidence beyond performance on the optimization data.

Critical weaknesses
  1. The validity of some metrics is unclear. Output-token-length variance is used as a proxy for comparable reasoning effort across languages, but tokenizer differences across languages can affect token counts, so this metric may not reflect reasoning effort. Answer consistency also depends on judging semantic equivalence across languages and accounting for cultural differences; the paper should explain the assessment procedure and its reliability more clearly.

  2. The claims about reasoning mechanisms are stronger than the evidence supports. The reasoning-unit analysis shows changes in output structure after prompt optimization, but it is primarily correlational. These changes may also be affected by shorter prompts, task type, or errors in the automatic classifier. Thus, the claim that optimized prompts induce more structured reasoning should be treated as an interpretation rather than an established mechanism.

Questions for authors

see weaknessessee weaknesses

3 comments
AuthorsOpenReview ↗

Official Comment by Authors

We sincerely thank the reviewer for the thoughtful and constructive feedback. Below, we respond to each point and include several additional results that we hope will clarify your concerns.


W1. The validity of some metrics is unclear. Output-token-length variance is used as a proxy for comparable reasoning effort across languages, but tokenizer differences across languages can affect token counts, so this metric may not reflect reasoning effort. Answer consistency also depends on judging semantic equivalence across languages and accounting for cultural differences; the paper should explain the assessment procedure and its reliability more clearly.

Thank you for raising these concerns! We would like to clarify the motivation and implementation of these two metrics.

Output length variance. We agree that tokenization efficiency varies across languages. However, our metric is intended to measure computational reasoning effort, rather than the amount of semantic content generated. For a given model, each generated token incurs approximately the same computational cost under comparable decoding conditions, regardless of language. Thus, token counts provide a meaningful proxy for inference compute spent before producing the final answer. While better tokenization may allow some languages to express the same reasoning more efficiently, this is precisely why we measure computational effort rather than semantic length.

Answer consistency. We would like to clarify that all benchmarks in our study have extractable and verifiable answer labels, so our consistency metric does not rely on subjective semantic-equivalence judgments or LLM-based evaluation. Specifically, we extract the final answer from each response and compare the resulting labels across languages. For example, in multiple-choice questions, consistency simply measures whether the model selects the same answer option across languages. This makes the metric objective, deterministic, and reproducible, without requiring judgments about linguistic or cultural equivalence.

We will also make this evaluation procedure more explicit in the revision.


W2. The claims about reasoning mechanisms are stronger than the evidence supports. The reasoning-unit analysis shows changes in output structure after prompt optimization, but it is primarily correlational. These changes may also be affected by shorter prompts, task type, or errors in the automatic classifier. Thus, the claim that optimized prompts induce more structured reasoning should be treated as an interpretation rather than an established mechanism.

We agree that our reasoning-unit analysis is correlational and should not be interpreted as establishing a causal mechanism. Our primary objective is to characterize how observable reasoning patterns differ across system prompts, rather than claim that these patterns directly cause performance improvements.

We would like to highlight two aspects of our existing analysis. First, we examine reasoning distributions separately across benchmarks (Figure 5a and Appendix Table 14), explicitly showing that the observed changes are task-dependent. We also acknowledge in Section 6.2 that shorter and cleaner optimized prompts may contribute to these shifts. Second, to assess classification reliability, we independently reclassified 15,000 randomly sampled responses using LLaMA-3.1-70B-Instruct instead of Qwen3-14B, obtaining Cohen's κ = 0.528 (Appendix A.13). This indicates moderate agreement, while also highlighting the limitations of automatic reasoning categorization.

We will revise the wording throughout the paper to frame these findings as associations between prompt optimization and observable reasoning patterns, rather than established reasoning mechanisms. Importantly, these interpretive limitations do not affect our primary experimental results demonstrating improvements in multilingual task performance.

AuthorsOpenReview ↗

Follow-up on our rebuttal

Dear Reviewer XwXD,

Thank you again for your valuable feedback! We hope our responses and additional experiments have adequately addressed your questions. We would be happy to address any further questions or suggestions you may have. If you feel your concerns have been resolved, we would greatly appreciate your consideration in updating your assessment. Thank you again for your time and thoughtful review!

AuthorsOpenReview ↗

Second Follow-up on the rebuttal

Dear Reviewer XwXD,

As the discussion period is approaching its end, we wanted to check whether you have had a chance to review our rebuttal and whether any concerns remain. We would be happy to provide further clarification if needed, and would appreciate your consideration of an updated rating if our responses have addressed your concerns. Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.