- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies if a model can recover all the information to get a complete spec using conversations. They introduce a benchmark, OptRecover, to perform this task. They evaluate on latest models and introduce Knowledge-grounded representation as well as a harness for performing the task. Findings include that current models cannot do this task reliably.
Critical strengths
I really like the idea of the work. Humans are usually vague in their specifications. Most of the progress is made by LLMs on math tasks because they are well specified. Hence this task of seeing if the LLMs can recover the correct specification is super interesting and useful.
The work evaluates on several new methods and tries to evaluate different stages using a fixed representation (although this could be a drawback as well)
Critical weaknesses
There are missing details in the draft on how the benchmark was constructed. such as: who are the experts who designed the optimization questions?
The gold answers were bootstrapped from the same parser that later scores model outputs.
There is no information on inter-annotator agreement and no count of how much experts changed the parser drafts.
The paper simulates users using LLMs, it would be better to have some human evaluation.
I strongly believe that the paper is very narrow although it had a potential to evaluate something better.
The measurement itself isn’t validated well enough to trust the rankings of the models. there are no statistics. Since the users are simulated, do we know how well are do they answer? If they even answer correctly?
There is no evaluation on how the scores change with different budget. Since the paper claims that they can do this at a budgeted setting, knowing this is super important.
The paper is also very hard to read with a lot of things blocks etc.
Questions for authors
Why is the natural language quesiton only based on shouldn't it be based on all the collected information? i.e. all question and answer pairs? I would imagine that the answer to previous question would change what i need to know next?
Why do we have business experts? you didn't evaluate with them either.
You say consolidated specification is reviewed by AI reviewer but how does the AI reviewer know if this is complete or not?