- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes a framework for choosing the right type of action to proceed when an agent faces an uncertain decision. It argues that an uncertainty score is insufficient to make the correct decision in practical scenarios, and follow-up actions among escalation, retry, or search should be properly distinguished. Toward this objective, the paper develops a mathematical basis for calculating the probability / value of each type of action under a scenario, and implements SCOPE to enable fine-grained control over these decisions.
Critical strengths
-
The paper provides rigorous mathematical reasoning to ground a quantitatively the scoring of each action type, which resolves possible ambiguity of how success / importance of each action type might be defined.
-
The paper tackles a practical problem: the agent must decide the right next action type before spending irrelevant compute on it, and it presents a framework with concrete implementations.
Critical weaknesses
- The paper's introduction is hard to read without sufficient context. The context is not hard to grasp -- it is just a decision problem when the agent faces a blocker in the trajectory. However, the paper makes the introduction so dry that even this point is buried in technical-sounding text.
- The method section is not well motivated. The equations are not connected properly to the introduction, nor to the proposal of the framework SCOPE, since not most of the equations even appear again in SCOPE.
- The utility of the framework is limited, since it is not compared with a prompting baseline which just asks the model to be aware of the alternative choices and think more carefully.
- No code is supplemented to verify the concrete implementation or the reproducibility.
Questions for authors
- Can you give the actual code or pseudocode that implements every component of SCOPE?
- Can you give at least two case studies that include actual trajectories from the agent?
- Can you report the baseline performance with just careful prompting the agent to be thoughtful?