- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper focuses on the problem of whether LLM assistants should ask clarifying questions or just act and let the user make corrections. The authors argue that keeping asking clarifying questions until the ambiguity is lower than a certain threshold (like prior works) is not optimal. A better LLM assistant should also consider the cost when raising clarifying questions. To this end, they proposed REVOIR, it first generates several user intent hypotheses then estimates the value of information (VoI) of candidate questions as well as acting. In the end, it decides whether asking is better or acting by also considering the cost. They evaluate their method on CondAmbigQA and ADAPT, showing that REVOIR is able to achieve higher success while reducing the number of clarifying questions.
Critical strengths
- It's a less studied problem but very practical issue for recent AI agents applications.
- Extensive evaluation across different LLMs detailed analysis. (Different LLMs and multiple evaluation settings)
Critical weaknesses
- There are two places in the paper that utilize a LLM to simulate a user. One is for evaluation the other is the internal user model in REVOIR (for scoring question candidates). For the former one, I've checked Table E.1, it seems like for "Human: not satisfied", the alignment is worse. For the latter one, I cannot find human evaluation. In the manuscript, the author claims that the internal user model and the external user simulator differ. But how does it differ? Without the human alignment evaluation on both the internal and the external user model, it is hard to make sure they are close to real applications.
Questions for authors
- It seems like in the setup of this paper, the "cost" is counted by the number of "questions/clarifications. However, this is somehow misleading. There can be different numbers of words/tokens or different amount of information it carries. Is there detail analysis on this?
2 comments
Thanks for the authors detailed response. The clarification makes the paper clearer.
Response to Reviewer Uiu315
Thank you for your positive review and the appreciation for our work. We address your concerns below.
REVOIR configs spend 2.1k–4.3k tokens (1.4k-3.1k input, 0.7-1.2k output) per example and parallel branch, similar to REIGN (1.5k–4.6k total, 1.0k-3.4k input, 0.5k-1.1k output) and ReAct (2.6k–5.7k total, 2.3k-3.5k input, 0.3k output) (see Tables R4 & R8).
Per example, REVOIR configs spend 10.6k–21.5k tokens (7.0k-15.6k input, 3.5k-5.9k output), REIGN spends 7.4k–22.9k (5.0k-17.1k input, 2.4k-5.5k output), and ReAct spends between 2.6k–4.4k (K=1, 2.3k input, 0.3k output) and 73.9k–113.3k (K=20, 17.3-67.1k input, 1.6k-6.9k output). Token costs increase with REVOIR / REIGN word budgets (see Tables R2 & R6).
In the measured range, REVOIR’s performance scales considerably better with more tokens, both per example and per branch, as estimated by robust linear regression (i.e. Kendall-Theil linear regression).
Under agent termination, REVOIR requires only 7.7 additional tokens for a gain of 0.01 in AnswerCorrectness, vs. 38.5 additional tokens for REIGN, and 33.3 additional tokens for ReAct (see Figure R1(a)).
Under user termination, REVOIR requires 50 additional tokens for a gain of 0.01 in AnswerCorrectness, whereas REIGN and ReAct decrease in performance with more tokens, dropping 0.01 in AnswerCorrectness with 24.4 and 83.3 additional tokens respectively (see Figure R1(b)).
Similar trends hold when regressing AnswerCorrectness against branch-normalized token usage (i.e. serial reasoning depth) (see Figure R2), with REVOIR either out-scaling or matching REIGN and ReAct. Under user termination, REIGN and ReAct decrease in performance with more tokens per branch, unlike REVOIR.
Together, these results imply that REVOIR scales more effectively with additional reasoning tokens than baseline methods, albeit with a higher minimum token usage per example.
Thank you again for the thoughtful review. We hope that our responses and revisions have comprehensively addressed your queries, and that you will consider raising your score.