- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper study at a very important research in LLM safety, which is that how can a model tell whether someone is asking an innocent question or just want to to get harmful information and guidance on dangerous things. For example, asking how locks work is reasonable at the first galance, but the same user question could becomes quite harmful if earlier users messages focused on breaking into a particular building.
To solve this problem, this paper propose Dialogue Critic-Guided Sampling. DCGS works in two stages. First, the model generates several candidate interpretations of the user’s question/intented. And a trained scoring model helps to select one candidate response by measuring how well the conversation would go if LLM take that interpretation; (ii) Based on the estimation, the model generates several final replies, and another models helps choose a response.
For the experimental results their proposed method show clear improvement to multiple kinds of attacks.
Critical strengths
This paper studies a quite meaningful safety problem. This paper pointed out that user's harmful question can become clear after the conversation with llm goes on after several turns. Based on this good intuition, the authors study that how the dialogue could be a reasoanble way to address both intented jailbreak attacks and unnecessary refusals of legitimate requests that ask an innocent questions.
Besides, the evaluation is comprehensive. Specifically, the users take both usefulness and safety into consideration. This paper measures both attack resistance and completion of benign requests,. Additionally, they also examine for various length of conversations, and tested on different model families. Generally, the evaluation is sound.
Critical weaknesses
My biggest concern is that for the current experimental results, it is not easy to determined whether the proposed approach, i.e., the two-stage intent modeling, actually drives the improvement or not. Specifically, their method combines several components: (i) candidate intent generatio, response reranking, etc. However, this paper haven't conduct abalation study clearly. My suggestion is that the authors could isolate the response spearately. This abalation study could help to decide it is which component that really carries the improvements. To sum up, more abalation is needed.
Besides, I think the theoretical is a bit of overclaimed. To be honest, I didn't fully understand the theorem. From my understanding, the theorectical results does not gurantee that better safety just because critic score is higher.
I also struggle to see why the proposed regret-based formulation should improve robustness to unknown user intent. It adjusts rewards by removing the lowest value obtained across the sampled intent hypotheses. For a given candidate pool, however, this shared offset leaves the ordering of immediate rewards unchanged (right?). I think the paper needs a clearer argument connecting this reward adjustment to its claimed robustness against unseen intend in my opinion.
Questions for authors
Much of the interactive evaluation uses a simulator that gradually reveals a predefined goal, and Figure 2 only shows five turns. Although the paper also includes SafeDialBench, the results leave open how well the method handles much longer conversations or attackers that specifically try to mislead its intent inference.