- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
(I think) the paper puts forward theoretical considerations that I would place close to work in abstraction to establish whether for every proposal made by a model there is an intervention that's robust with respect to uncertainty. In a nutshell the key point put forward is the limitations implied by a state-only harness.
I appreciate in general the line of research here pursued and I think there is merit in these observations. However I have concerns in terms of:
- The clarity of the contribution made;
- Some of the experimental results (see below)
Critical strengths
Analysing the limits of state-based harnesses in history dependent models and traces is I think novel and potentially significant. The machinery put forward seems to provide a good leverage to understand the resulting properties.
Critical weaknesses
I have concerns about the formulation and clarity of the results. These concerns start from the abstract. I would encourage the authors to be much more clear in terms of what is being claimed and avoid negative sentences eg in the abstract to denote their contributions.
While I appreciate the general direction of research, it seems to me that at a higher level the paper states that for history-dependent runs a state-based harness inevitably will be insufficient in some key properties. This is consistent, even if not nearly directly following from, results in computation semantics of history-dependent processes and results in run-time monitoring including automata theory (which is unconnected to LLMs but related in this context)
Experimental results. Appendix B. It seems to me that full and no effect verification execute the same operations against the same environment trace until their first no-commit. Therefore the number of full-harness retries (in their terminology) should be fewer than the number of no effect verification retries that fail * number of retries. But in the table (page 24) this is not the case. What am I getting wrong?
My concerns above are conditioned by limited expertise on the subject matter.
Questions for authors
Appendix B. It seems to me that full and no effect verification execute the same operations against the same environment trace until their first no-commit. Therefore the number of full-harness retries (in their terminology) should be fewer than the number of no effect verification retries that fail * number of retries. But in the table (page 24) this is not the case. What am I getting wrong?