- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
As AI systems are increasingly deployed as "agents," able to engage in multi-step reasoning, whether these intermediate choices are fair becomes all the more important. The authors develop an auditing framework of multiple steps of agents to assess their fairness. They then develop a scaffold to improve fairness which they call PRISM.
Critical strengths
I think the idea of evaluating the fairness of intermediate agentic choices is very clever and important. I think the problem the authors take on (and the initial framing in the intro) is very good, and I learned something there!
I also generally think the approach the authors take with PRISM in separating criteria formation and case eval is sensible, and I appreciate that the authors engage in both eval and productive scaffold design.
Critical weaknesses
Unfortunately, I really struggled to understand the heart of the paper and the results. I found Section 3 and 4 admittedly quite confusing. I'm not sure if this was from AI writing (though I suspect that was a substantial part), or my naivete (also possible, and I've flagged lower confidence from myself in the review!)
I do appreciate the authors had key takeaways! Though some like "evidence boundary" are confusing and not quite paper-publishing-ready, in my opinion. For instance, that one says "supported" (but it's not clear the specific hypothesis?)
I think the body of those sections would benefit from a more substantive rewriting.
The appendix is also extremely hard to follow. Again, I think this is substantively AI-generated.... which seems to lead to quite confusing and unclear structuring. At minimum, adding a table of contents would help...
Questions for authors
I loved the idea of assessing the fairness of tool use. It could be worth bringing some qualitative exploration of tools used into the main text? (say for track B) that could help with clarity and grounding the work for readers!
Please check and acknowledge the author response to your review
Dear reviewer — as the discussion phase is coming to a close, please acknowledge and respond to the author responses to your review. It would be helpful to provide time for the authors to comeback with further follow up if necessary.
Thanks AC