- Rating
- 1(−1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents 48LawsBench, which assesses whether large language models would recommend means to manipulate or harm third parties when providing strategic advice based on the "48 Rules of Power", and distinguishes between three scenarios: explicit requests, implicit confrontations, and neutral targets. Overall, the topic selection of the paper is of practical significance, the reasoning is clear, the writing is easy to understand, and the experimental coverage is relatively wide. However, the effectiveness of the evaluation indicators and the generalization ability of the mitigation methods still need more thorough verification.
Critical strengths
- The topic is of practical significance and offers valuable insights for the security assessment of large language models.
- The paper demonstrates sound overall logic and a clear structure, with the research motivation and key contributions explicitly stated.
- The experiments cover a variety of models, and the analysis of the results is comprehensive.
Critical weaknesses
- The evaluation relies heavily on automated scoring, and the reliability of the results requires further verification.
- Comparisons with other methods and ablation studies are insufficient, and the sources of the key improvements remain unclear.
- The experimental scenarios are somewhat limited, and the method's effectiveness in broader applications remains to be verified.
Questions for authors
- Provide results regarding the consistency between human evaluation and automated scoring, and clarify whether the main conclusions remain robust when the scoring model is changed?
- Include a comparison of methods under the same inference budget, as well as ablation studies on key modules, to pinpoint the source of the performance improvements?
- Does the method remain effective across a wider range of models, diverse scenarios, and situations where target rules are not pre-specified?
4 comments
I compared the current version with the original submission and found no changes in the main text or appendices. I also could not find the results of the additional experiments mentioned in your response. It remains unclear how closely the automated LES and GUS agree with human ratings, and whether GreeneGuard outperforms the iterative-refinement baseline under the same inference budget. Given how directly these questions affect the paper’s main conclusions, I am lowering my rating from 2 to 1.
We would like to clarify a misunderstanding: we have not yet submitted our revised manuscript or the results of the additional experiments.
Our previous message was intended as a preliminary update on the work underway, and to solicit your feedback before finalizing the revision.
The scope of the additional experiments and analyses has been substantial, which is why we have taken additional time to complete and carefully validate them. We are now finalizing the revised manuscript and rebuttal, and will submit both shortly.
We would be very grateful if you could reconsider your assessment after reviewing the forthcoming revision, as the currently available manuscript does not yet reflect any of these additions.
We thank you for your constructive feedback. We have revised the manuscript extensively, addressing all the comments through additional experiments, analyses, and clarifications.
New or substantially revised content is highlighted in yellow, with margin tags indicating the corresponding reviewer and comment (Q = Question, W = Weakness). To accommodate the additions, we slightly condensed some existing text.
W1. Reliability of automated evaluation
We introduce five human-validation stages (Section 2.4, Figure 2, Appendix D), alongside extensive judge-robustness analyses (Section 4.4, Appendix F.3, Figures 13–23).
Q1. Human versus automated scoring
We report human–automated scoring agreement in Appendix D.3, with additional human validation results in Appendices D.1, D.2, and D.4. The corresponding questionnaires are provided in Appendix D.6.
Q1. Robustness to changing the judge
We address this in Section 4.4 (Figure 4) and Appendix F.3 (Figures 13–23). The tested cross-model ordering persists across judges, despite differences in their absolute scores.
W2. Comparison with other methods
We add a comparison with Self-Refine in Appendix F.4.6 (Figure 28), evaluated under two judge settings. GreeneGuard more effectively suppresses high-LES responses for the most problematic laws while preserving high GUS.
W2/Q2. Ablation studies.
We substantially extend the ablations in Appendix F.4 (Figures 24–28), including analyses of judge reasoning effort, model size, model generation, output length, and baseline instructions.
We further discuss the underlying patterns in Section 4.5 (p. 9) and Appendix F.5 (Figure 29).
Q2. Equal inference budgets
Appendix F.4.2 (Figure 26) adds a response-length sensitivity analysis where we force the answer length to be between 150-200 tokens.
We also analyze inference costs in Appendix F.4.3 (Tables 6–7) and in Appendix F.4.6 (Figure 28) where we investigate GreeneGuard and Self-Refine.
W3/Q3. Broader scenarios and unspecified target laws
We broaden the evaluation across models and settings (Appendices F.3–F.5, Figures 13–29). More importantly, Appendix F.6 (Figures 30–31) introduces a multi-law GreeneGuard extension that identifies and mitigates active high-risk laws without specifying them in advance for each scenario. Across 20 scenarios combining multiple laws, 13 responses satisfy the joint risk and utility criterion.
Thank you for your detailed feedback! We have been working intensively on strengthening the paper along all three directions you identified:
W1/Q1 (Evaluation reliability): We expand human validation of LES/GUS scoring, rescore all evaluated models using a common judge, and conduct cross-judge calibration on identical responses. We also add independent PSRL validation with two human raters and a separate two-reviewer appraisal of advice actionability, willingness to follow, and expected effectiveness.
W2/Q2 (Comparisons and ablations): We add comparisons against an alternative iterative-refinement baseline, and evaluation with fixed-length model advice (to control for differences in advice-generation length and its associated output-token costs). We further investigate reasoning effort, model size and family, provide more detailed results for differences among prompt safeguards, and we analyze execution traces in more detail to clarify where improvements originate.
W3/Q3 (Generalization): We extend GreeneGuard with automatic identification of relevant high-risk laws, removing the requirement that target laws be specified in advance. We additionally evaluate the resulting system on mixed user traffic, including requests entirely unrelated to strategic advice, legitimate advice, and high-risk advice.
Could you please confirm whether these additions constitute the comparisons, validation, and generalization tests you had in mind? In particular, would you require any additional specific baseline or component-level ablation beyond those described above?
If we include these additions in the revision, would you consider revising your recommendation toward acceptance? Your feedback on any remaining methodological requirement would be extremely valuable before we finalize the revision.