- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces KARMA, three rules for deciding which agents and sources to trust. It tests these rules as prompts on synthetic credibility tasks and as a hand-designed Bayesian reputation system. The prompts help several mid-tier models, while the frontier model is already at ceiling.
Critical strengths
- The paper reports all the prompts and full details of the simulated data used in the paper. The paper also reports important negative results.
- The findings are informative that the prompts help several mid-tier models, while the frontier model is already at ceiling.
Critical weaknesses
- The experiment is a stateless source-selection task rather than an interacting multi-agent system. KARMA prompt directly states the decision rules used to construct the benchmark. The 150 scenarios are variants of only 6 logical templates, and the extreme benchmark was constructed after the pilot was used for model selection and difficulty calibration. Thus, the large gains can show that explicit task-specific rules help selected models on scenarios constructed around those rules, but they do not convincingly prove general epistemic robustness or improved multi-agent collaboration.
- The powered benchmark lacks direct ablations of P1, P2, and P3. The paper should better isolate the claimed contribution: one thing is to distinguish KARMA’s training-free packaging and operationalization from more direct methods like direct evidence, unreliable recommendations, distributed trust, etc.
Questions for authors
Can the principles be evaluated in an actual multi-agent workflow where credibility judgments can practically affect task delegation, information propagation, final-answer accuracy, or system utility?