- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper measures how much an LLM's answer in an actor-critic setup changes when only the role wording of the prompt changes. It uses 2000 multiple-choice items under nine prompt variants, and finds that eight instruction-tuned models (0.5B to 14B) are all sensitive.
Critical strengths
The question is relevant, since role prompts in multi-agent pipelines are rarely ablated.
Critical weaknesses
W1. The paper is very hard to follow. The task is never properly introduced, background is assumed, and the text relies on a large set of coined terms that are defined late, loosely, or not at all. Some examples:
- The paper never states clearly what a "collaborative system" does at evaluation time. The abstract and introduction talk about actor-critic pipelines. Only in the "Endpoint" paragraph of Sec. 3 does the reader learn that every system is scored by a single forward pass on a single prompt.
- Three near-synonyms are used for the prompt wording around a question. "Rendering" (Sec. 3) refers to the nine evaluation prompts, "frame" (Sec. 4) to the four training prompts, and "role interface" is used for both. The reader has to work out that these are the same kind of object.
- Further terms are introduced with little or no definition: "endpoint", "anchor", "harmful revision", "trap families" and "answer contract", "mean pairwise view dissimilarity", "Protocol-Mix".
- The pair-mining procedure is described only by reference ("we follow the trajectory construction of ACC-Collab"). The "guided versions", "steered toward y and away from it", are never explained. It is not stated how the steering is done.
- Figure 2 packs three separate diagrams into one and is hard to parse without first reading Sec. 4. The rest of this review reflects what I could reconstruct, but I think the paper needs a substantial rewrite before it can be properly assessed. I would be glad to review a revised version.
W2. he paper relies on what the authors call "a fixed answer cue" and that in the field is known as "structured response". This is known to affect the response of the models, and alter them[1], which I believe to unfotunately invalidate the results obtained by the authors.
W3. The method is trained and analysed on one model, Qwen2.5-3B, with pairs mined from its own trajectories. The cross-model evaluation stops at 14B and includes no reasoning models. Because the endpoint allows no generation, the benchmark cannot evaluate reasoning models at all, even though current actor-critic systems are mostly built on them. The evaluation is also is limited to only 2000 samples, from a dubious mixture of benchmarks.
W4. The base model's accuracy on the six neutral prompts lies between 69.3 and 70.1 (Table 6). The 10-point spread therefore comes almost entirely from the "overconfident" prompt. The volume control alone, with one interface and four times the steps, already brings the spread to 2.13 and worst-case accuracy to 68.74, against 1.49 and 69.68 for the full method. Variety adds +0.6 average accuracy and +4.1 agreement, of which 1.07 is due to prompt length (App. L). It seems that the gains are rather small, and maybe not coming from the proposed method. More testing would be necessary to quantify the improvement.
[1] Xinpeng Wang, Bolei Ma, Chengzhi Hu, Leon Weber-Genzel, Paul Röttger, Frauke Kreuter, Dirk Hovy, and Barbara Plank. 2024. “My Answer is C”: First-Token Probabilities Do Not Match Text Answers in Instruction-Tuned Language Models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 7407–7416, Bangkok, Thailand. Association for Computational Linguistics.
Questions for authors
See above