- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper aims to present release-assurance framework for multi-agent LLM systems dubbed PCG-MAS, which ams to provide assurance to a downstream users about the system’s outputs. They evaluate seven model families on eight datasets spanning factual reasoning, web interaction, function calling, and adversarial agents. The authors claim improvement on several metrics for multi-agent systems.
Critical strengths
None
Critical weaknesses
The paper has a main drawback in its disjointedness between the narrative, background and explaining results.
The formalization of the paper has multiple weaknesses. Some of the key terms are not defined, or are loosely coupled with other terms in the main text.
First, the paper does not precisely define the “release-assurance framework” which is the key topic of this paper. While a reader can discern what it may mean from the narrative, this point should be fully clear.
Figure 1 is fully out of place. It cannot serve as a summary of the framework and the contribution. Most of the terms in Figure 1 are not defined, or are defined only later in the paper. As such, a reader cannot get a proper understanding of the results. This figure should be presented only in the Evaluation section.
“Release stream” is not defined before use in the introduction. Same with “committed epochs”. The scope of the audit is not clear until Section 3, and even there the definitions are never explicitly connected.
Evaluation is not providing clear insights, just numbers. The evaluation scope in the introduction is just a laundry list of models and settings that were evaluated, without a clear insight of the results. Similarly, the contributions are failing to clearly state the novelty of this work for the broader ML community.
The definitions in Section 2 also suffer from “impatient formalization”, which fails to describe all the terms in the formulas before they are being used. Let us take the definitions 1 and 2 for example
D1:
- ‘Typed’ in the typed graph is not clearly defined
- Evidence nodes carry information that is not defined, neither the functions h and H or the function x have been defined at this time.
- Other terms, E, \tau, attracted are not being defined.
D2:
- The details of the consumer contract are not given
- consumer scope is not properly defined in D3 (it just refers to the opaque Derive_\Omega term).
- Producer witness and all remaining terms are either not defined at all or are defined much later in the text.
In either case, the missing parts of definitions make the remaining formalization difficult to understand and appreciate.
The evaluation is similarly difficult to comprehend. It just lists various statistics without putting them in the context. Each RQ is description contains only a 2-3 sentences, that neither explain the setup, the form of the plots/tables nor the insights. The choice of benchmarks and the metrics are also missing a clear justification.
Finally, the related work is missing from the paper and fully relegated to the appendix. This also makes it hard to understand the novelty of this paper compared to the state-of-the-art.
These sections confirm the underlying issue of the paper: it aims to put together too many concepts and results, that it fails to explain any of them clearly. As a result, the paper is not self contained.
Overall, the paper is not ready for publication at this time.
Questions for authors
What is a precise working definition of release-assurance framework used in this paper?
Several plots have the labels “Derived”, “Direct”, and “Protocol”. What is the meaning of these labels and how they differ?
Can you provide the justification why these benchmarks were selected?
Last 24 hours of the reviewer discussion period
Dear Authors and Reviewers
Just a reminder we are soon to be entering the last 24 hours of the reviewer discussion period.
Authors: If you have not yet responded to the reviews and intend to please do so ASAP. Please note reviewers will most likely not have time to respond to any late rebuttal comments so please make sure any comments are clear and not in need of further clarification.
Reviewers: If the authors have responded to your reviews and you have not yet responded please do so, even if it is a one line response acknowledging you have read the rebuttal.
Thanks for all your time, and I hope you all learnt something interesting during the process.
Kind regards,
Your ICLR Area Chair