- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper aims to produce a better optimization method for improving the rank-based verification capabilities of LLMs than something done in post-hoc analysis. The author/s motivate this by citing that being able to verify outputs as evidence is necessary for improving an LLM’s reasoning and generation capabilities for decision-support tasks. The paper evaluates its proposed Evidence Coupled Policy Optimization (ECPO) method on only two open-weight models and on two datasets (MAVEN, RAMS) for the entire paper.
Critical strengths
I tried to find an angle where I can appreciate the paper more but the content and writing of the paper itself is very unclear and confusing which makes it hard for readers like me to appreciate its contributions. I guess I appreciate the authors’ efforts towards putting together a proposed optimization method that aims to improve verification consistency of LLMs but the corresponding experiments, writing, results do not support this goal coherently in the paper.
Critical weaknesses
One of the main critical issues with the paper is that it is highly unreadable due to the written content being very AI-sounding and the overall writing of the paper could be further significantly improved in quality and readability. I acknowledge that the authors did mention they use AI for the paper and take responsibility for it, but I still highly encourage the authors to edit out any AI written text in the main paper given that they are very poorly written and are obvious to readers. The paper is also full of uncontextualized technical jargons such as “finite window-local roster” “frozen event extraction” “four-stage event skeleton” “auditable support contract” “source-normalized labels”, etc. which could have been rewritten in very crisp and simple language instead. It’s very hard to grasp what the authors wanted to say and hard to differentiate which were the authors’ main idea vs. what has been AI generated from this work.
The paper has no substantive and in-depth discussion of the results. The content written for Section 6 barely makes any sense to the reader and does not make it clear whether the proposed methods are responsive to the research question. Basic simple questions remain unanswered such as “What method per model works?” “Which model is better at what particular test dataset?” “What metrics actually matter for the task?” “How do the methods tested fare against simple and stronger baselines such as through prompting off-the-shelf LLMs like GPT-Sol?” “What type of models have better performance?” “What error analysis exposes the selected models weaknesses for the task?” These are very simple questions you would expect to be discussed thoroughly for an ICLR paper but these are completely missing and not properly and clearly written for the current submission.
The paper uses two types of models for its main results: Llama-3.1-8B-Instruct and Qwen3-14B. There is also no baseline comparison. These two are also grounds for clear rejection of the paper for lack of technical rigor and does not meet the empirical standards of ICLR. I highly and strongly recommend the authors to read previously accepted papers from ICLR/NeurIPS/ICML to get a level of how rigorous technical experiments are conducted and accepted in these venues. Simply running your methods on two models is simply not enough, much less selecting commercial models.
Questions for authors
N/A