- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes a framework to analyze runtime formal verification of agents with fallible observation of the atomic propositions, with potential application to GUI agents. It derives the soundness and completeness probabilities of monitors (modeled as conjunctions of atomic propositions) under assumptions of independent observation of atomic propositions. The framework includes modeling of the utility of including the monitor and derives a critical threshold for net positive utility. It applies this framework on existing monitoring mechanisms and on the dataset CON-150.
Critical strengths
- The problem (runtime verification of GUI agents) is very interesting and of high value if properly solved. The proofs seem correct to me.
- I'm not very familiar with this field, the underlying theory and proposed monitor architecture might be very important despite being somewhat buried in the poor writing.
Critical weaknesses
- The presentation and writing make the paper very difficult to read as it is. Many non-standard jargons and terms are used with little context or proper definition. The experiment setups are not clearly presented and the relevant appendix sections are not pointed to in the main text.
- The case under consideration seems very simple, namely the monitor being a conjunction of atomic propositions that can be read off from the input by the fallible sensor (grounding operator). This makes much of the theoretical derivation almost tautological. That being said, I do appreciate the value of the framework itself.
- The connection to GUI agent seems rather indirect to me. The whole framework seems to be just condering a fallible sensor (grounding operator). The empirical part directly compares with Kamath and Xavier, none of which seems to have been applied on agents with vision modality verification. I cannot identify the dataset used, CON-150, throughout the paper, which seems to be the only real dataset mentioned in the main text. I believe this is the proposed dataset because of the primary area choice. But there is very little description about this dataset itself.
Questions for authors
- What is CON-150? Does it invovle visual modality?
- Is figure 2 measured or predicted/simulated?
- Is it fair to assume monitors being conjunction of atomic propositions, with no logical dependency between them? I'd imagine there can be atomic propositions with nontrivial logical dependency while one cannot reduce any completely to the rest.
- Could you please clarify where the detail of each experiment presented in the main text can be found in appendix, if at all? Could you please clarify how the content of each appendix contributes to the whole paper?