- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces HarnessSHAP, which attributes outcomes of agents to each trajectory units. Authors conduct experiments on three benchmarks and compare XAI-Grad against fixed harness and generic textgrad.
Critical strengths
- The topic about credit assignment for trajectories for debugging or auditing is timely and important
- The set up is somewhat promising, e.g., the context are restrict to prerequisite-closed units.
Critical weaknesses
- I find results in Table 1/2 very suspicious: the three benchmarks increase in almost equal steps, i.e., drop rises by +0.04, validity by +0.06/0.07, top3 by +0.06. These numbers seem to be generated by LLMs instead of actual results. If I am wrong, please correct me with CIs and variances.
- The experimental setup is very unclear, e.g., the model choice of agents, judges, etc. are not even revealed, the number of tested trajectories are not reported. The presentation is also suboptimal. For example, Table 3 does not specify which benchmark it uses.
- The baseline selection is not fully fair. For example, a more related baseline AgentSHAP is completely ignored. Seems that all comparison are based on prompt variants of the paper's pipeline.
vspaceis used too aggresively, the table captions are very close to the tables. And why some captions are placed under the tables? I do think this is actually allowed. Moreover, the font size of those captions which are under the table is noticeably smaller than required (e.g., l.191-195).
Questions for authors
see weakness
Response to Reviewer eHXG
Thank you for the direct review. In short: (1) the revised Table 1 reports variability across five seeds and corrects the call counts. (2) We will specify the setup in full. (3) We clarify the relation to AgentSHAP. (4) We will correct the formatting.
1. Results in Tables 1 and 2.
Thank you for asking for variances. The revised Table 1 reports Drop, Valid and Top-3 as mean ± sample standard deviation across five seeds, with standard deviations between 0.006 and 0.021. For example, on ToolSandbox Drop is 0.36 ± 0.017 for the Fixed harness, 0.40 ± 0.007 for Generic TextGrad and 0.44 ± 0.012 for XAI-Grad. The submitted table also reported incorrect explanation call counts, and the revised table corrects them (general response). We apologize for this error.
2. Experimental setup and Table 3.
We will report the agent, judge, critic and editor models and the number of tested trajectories per benchmark in Appendix A, and state the benchmark of Table 3 in its caption.
3. Baseline selection and AgentSHAP.
AgentSHAP, which the paper cites, attributes outcomes to tools with Monte Carlo Shapley values over tool subsets. HarnessSHAP attributes outcomes to individual trajectory units, suppresses each unit together with its dependents and lets the agent regenerate the remaining trajectory. The two methods answer different questions at different granularity. AgentSHAP is the closest Shapley-based method, and we will add a direct discussion of the two.
4. Formatting.
Thank you for noting this. We will place all captions above the tables at the required font size and remove the spacing compression.