- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The current natural-language-to-TLA+ benchmarks may overestimate the correctness of the generated specifications, failing to capture the semantics mismatches between the generated and the reference protocols. So this paper proposes EQUINOX benchmark, which forcus on designing strict criterion for the generated specifications from agents. It applies bounded trace equivalence as the criterion, which will checks if there are ommited behaviours or extra behaviours between the agent output and the reference. The experiments show that 1. the proposed criterion will refuse most samples that could have passed execution checks, 2. the agents show similar distribution of success samples, 3. repeated attempts don't change the failures.
Critical strengths
- The paper addresses part of the evaluation correctness problems of the previous execution pass checks
Critical weaknesses
- The proposed method of checking the validity is a good improvement over the previous methods, but it still could not provide assurance on the validity. I still don't know how much is the difference between the real pass rate and the current pass rate.
- The experiments seems not comparing the proposed check method and the corresponding component of the previous benchmarks. The paper only compares the checks defined by the authors.
Questions for authors
- I wonder if the checking methods of current benchmarks are the same as the experimented ones (G1, G2, G3)?