- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper is a survey on the effects of various methods in the setting of generation against a fixed ontology. The three evaluation datasets are OOD-100, and Strict-20, each containing items encoded in IES4 and Biolink, respectively. The authors test and analyze a few methods, including a greedy generation baseline, a best-of-N sampling, trie-based constrained decoding, etc... and measure many metrics including syntax validity, membership rate, and the overall metric . The authors report a subset of these metrics on one dataset, and then go on to analyze the effect of a few methods and detail their limitations in the subsequent sections.
Critical strengths
None
Critical weaknesses
My concern is that this paper contains a large amount of LLM generated text. None of the datasets or metrics are explicitly defined in text (only using plausible sounding vocabulary such as syntax validity, membership rates, F_1 score). Moreover, there are many formatting errors and inconsistencies between the descriptions.
Major issues (a small subset).
- No (high-level) description of the task / any examples. What are the IES4 and Biolink ontologies? What are the OOD-100 and Strict-20 datasets? [103-107]
- No concrete description of the metrics. The metrics included involve "syntactic validity, membership, choice , and " [119]. How are each measured? The text mentions that all generations are possible as per some "system" [137], what is this system? What are the definition of some terms: "canonicalisation slack" [142] "entailment-reduced variant" [141]?
- Models. What are the "frontier arms [that] run three tiers of one commercial family"? [153] What is the "open 30B MOE coder model" used by the local family [155]. What is a "3B floor check model? [155] What purpose does it serve and where are the results of this floor?
- Methods. No description at all of the methods outside of a brief one line description. How are you forming the "plain validator loop" [169] or doing "constrained decoding" [170]. No details contained in appendix at all.
- Findings. The findings indicate that the best performing model is just a frontier model (where are the other tiers?) with the vocabulary in context? What should I take away from this?
- BoN sampling. Why is this a separate table? What intervention is this BoN on? [188]
- Biolink. Where are the results for this? These results are mentioned on line 353 but never shown in text or appendix.
- Formatting issues. All the section headers and titles break format convention.
Questions for authors
n/a