- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes CDSCD, a method for finding which ordered sequences of skills, tool calls and verifier steps in an agent’s execution carry a counterfactual group disparity, and for removing it by rerouting those sequences identically for all groups. It reports better localization than edge-level patching on synthetic graphs (AUPRC 0.81 vs. 0.74) and a drop in counterfactual flip rate from 0.143 to 0.061 at a 0.009 cost in task success.
Critical strengths
- Applying circuit-style interventions to observable agent events (tool calls, verifier states, fallbacks) is a reasonable idea.
- The paper includes negative controls and a condition where the method should abstain.
Critical weaknesses
- The notation feels needlessly complicated, but also hinders the reader from understanding and assessing the method. It is strange to define this Agent A object with so many auxiliary tuple components, then define another set of variables that are part of a trajectory, with no intuition for how all of these connect. What generates the public state summary? How are tool calls, verifier decisions, and branches represented within the trajectory? Also, the fact that the trajectory has some temporal nature to it makes me wonder why the authors didn’t just use the Markov decision process, or some other formal sequential decision making object that would be easier to follow. Some other questions in the method section that could use clarification: what is a “matched variant”? What is a “preregistered protected-attribute representation”? As a rule of thumb, it helps to have a running example to help explain your terms. Some other presentations comments: Fig. 1 doesn't communicate much of substance; Fig. 2 enumerates a lot of components but doesn't clarify many of these questions nor help the reader understand the method.
- I can't verify the experiments: the paper doesn't state how many base tasks it uses or what the protected attribute is (Appendix E uses literal GROUP_A/GROUP_B tokens), and it tests only two 7-8B models. The paper also looks carelessly prepared, with wrong bolding in three columns of Table 2 (for success and progress metrics), four unresolved "(?)" citations, and a power-systems paper cited for interchange interventions.
Questions for authors
See above.