- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies Procedural Exposure Bias (PEB), where repeated procedural content biases LLM agents toward particular actions regardless of their applicability. The authors show that representing skills visually can attenuate this effect and evaluate the finding through controlled experiments, mechanistic analyses, and three agent benchmarks.
Critical strengths
- Extensive experimental design. The controlled states, keyword-replacement experiments, and rasterized-text comparisons provide useful evidence for the observed bias.
- Comprehensive evaluation. The paper investigates multiple models and benchmarks, complemented by action-level and representation-level analyses.
- Interesting empirical observation. The difference between text and visual skill representations offers a potentially useful perspective on skill-augmented agents.
Critical weaknesses
- Unclear selectivity of visual attenuation. The reduced exposure bias under visual encoding may reflect a general weakening of procedural information's influence on action selection, rather than a selective mitigation of irrelevant exposure effects. The current experiments do not sufficiently distinguish these possibilities.
- Limited statistical support for practical improvements. Several end-to-end improvements are relatively small, and the main end-to-end results lack confidence intervals or statistical significance tests.
Questions for authors
- Visual attenuation: Does visual encoding selectively suppress irrelevant exposure-induced action preferences, or does it generally weaken the influence of procedural information on action selection? Could the authors provide controls involving relevant procedural information to distinguish these possibilities?
- Statistical significance: Are the end-to-end improvements statistically significant under paired evaluation?
4 comments
Dear Reviewer,
Thank you again for your valuable feedback. We hope that our responses have addressed your concerns and would greatly appreciate any further thoughts you may have.
We would be happy to provide additional clarification if needed.
Thank you for your time and consideration.
Feedback
Hi, the review deadline is approaching. Please take a look at the authors' rebuttal and post your feedback (and update your rating if appropriate) at your earliest convenience. Thank you! AC
Thank you for your response. However, it only partially addresses my concerns, and the experimental evaluation remains insufficient to convincingly support the main claims. Therefore, I will keep my rating unchanged.
We sincerely thank Reviewer jsqM for the thoughtful and constructive suggestions concerning access to procedural information, relevant-action exposure, and statistical support. We appreciate the attention to distinguishing the reliability of our controlled PEB findings from aggregate agent outcomes.
Reply to Weakness 1. The submitted Figure 3b shows recoverability of repeated procedural signals, while Figure 1 establishes weaker distractor-preference shifts under Visual. We extend these findings with Table R1, showing accurate complete-action recognition, and Table R2, showing a larger reference-action than distractor-action preference gain under Visual in the tested verb–object condition. Together, these controls support recognizable procedural content and differentiated responses to action relevance, reinforcing the interpretation of representation-dependent resistance to irrelevant exposure. They address the alternative explanation that reduced interference necessarily means that the model cannot recover or identify the relevant procedural content.
Reply to Weakness 2. The submitted Table 1 reports higher Visual macro-average task success across evaluated backbones. In addition, Tables R3–R4 provide paired confidence intervals for the central action-level exposure effect, and Appendix A.3.3 reports a paired bootstrap difference in executable-action exposure slopes. These analyses strengthen the statistical evidence for representation-dependent PEB, while Table 1 provides the complementary aggregate task-success comparison.