- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Matched Policy Intervention (MPI), a module-replacement experiment under strictly controlled conditions, to address action-space privilege and better analyze the relative strengths and weaknesses of systems. The paper applies MPI to conduct experiments and analyses on CausalGame, CARE, and a discrete-action adaptation of CausaLab.
Critical strengths
None
Critical weaknesses
- The paper’s contribution rests on the assumption that prior scientific research involves over-attribution, as illustrated on the left side of Figure 1. However, the paper does not examine how existing studies conducted their comparisons or establish the prevalence of issues such as action-space privilege to justify its motivation.
- In my view, MPI is not a novel scientific method but is essentially the controlled-variable method. Furthermore, the paper provides relatively few experiments, and its analyses are limited to comparisons between LLMs and statistical methods.
Questions for authors
None
4 comments
Response to Reviewer KPFr
2. Methodological contribution and additional experiments
We agree that controlled experimentation and factorial contrasts are established methods. The difficulty in applying them is that replacing an agent’s decision module can change more than its decision rule. Candidate generation, internal model state, tool outputs, and execution logic may be spread across the implementation, so the boundary between policy and support is not always clear. MPI addresses three practical difficulties.
First, a final score cannot separate policy quality from the quality of its support. A statistical routine may generate and score candidates while an LLM has to propose actions from raw observations. Alternatively, statistical computations may continue supplying recommendations after the LLM takes over. Repeating either comparison cannot separate factors that change together. MPI separates candidate construction and supporting computations from selection, then estimates policy effects by replacing the selector under shared support. In the support experiments, it changes menus while holding the policy fixed to measure the support effect.
Second, a shared dataset does not establish equal information at the decision point, and replacing a model does not establish a change in behavior. One routine may read fitted state and cached scores while another sees a textual summary. An LLM may also reproduce the statistical policy’s choices under the supplied support. Similar final scores cannot distinguish these cases. MPI checks whether the statistical rule can be reconstructed from the visible record and logs both recommendations at the same reached state. Reconstruction checks for information dependencies missing from the interface; policy disagreement measures whether the replacement changes behavior. Together, they help distinguish unequal information, identical choices, and different choices with similar outcomes.
Third, different recommendations need not produce different outcomes. LLM outputs may be repaired, retried, or replaced by a fallback, while another controller’s outputs are executed directly. Earlier choices also affect the evidence and opportunities available later. A complete-system score cannot locate the contribution of the original recommendation, the execution wrapper, or later stages. MPI fixes execution and failure rules, records recommendations separately from enactment, and preassigns execution authority across stages. The crossed design estimates each stage’s effect on the final outcome, including its downstream consequences. Confidence bounds then distinguish effects estimated to be small from effects that remain too uncertain to resolve, even when the policies behave differently.
None of these problems has to produce a software error. A reproducible benchmark gain can still answer the wrong attribution question. MPI makes the intervention auditable by specifying the boundary between policy and support, the matching requirements, and the stage assignments. This is the contribution of the protocol; the factorial estimator itself is established.
We added an exploratory acquisition experiment in a six-node CausaLab-style environment to test policies beyond the LLM and statistical pair. It compares a coverage rule, random selection, depth-two surrogate search, minimal tabular Q-learning, and fitted-Q policies using Ridge regression or a two-hidden-layer MLP. Under each support condition, all policies choose from the same four plan constructors and use the same final prediction module. The screened menu contains four distinct intervention-plan multisets. Training uses separate simulator seeds, and the MLP results average three frozen initializations. Evaluation uses 512 fresh test seeds under two support conditions, for 8,192 episodes across eight implementations.
| Acquisition policy | Concentrated menu | Screened menu | Screened − concentrated (adjusted 95% CI) |
|---|---|---|---|
| Coverage rule | 81.45% | 87.70% | +6.25 pp [+1.18, +11.32] |
| Random plan | 77.54% | 78.32% | +0.78 pp [−5.59, +7.15] |
| Depth-two surrogate search | 82.23% | 87.89% | +5.66 pp [+0.62, +10.71] |
| Tabular Q-learning | 79.10% | 85.94% | +6.84 pp [+1.09, +12.58] |
| Ridge fitted-Q | 81.64% | 87.30% | +5.66 pp [+0.69, +10.64] |
| MLP fitted-Q, three-run mean | 79.49% | 85.09% | +5.60 pp [+1.00, +10.20] |
The intervals use paired seed-level contrasts with Bonferroni adjustment over 21 specified comparisons. Five policy families show positive support effects; the random-selection contrast is inconclusive. Action-space construction therefore affects rule-based, search-based, and learned policies as well as the LLM and statistical policies in our main experiments. We chose the latter as cases where decision procedures and supporting computations can be intertwined, but MPI applies beyond those two classes.
References
-
Lin, B. Y., Fu, Y., Yang, K., Brahman, F., Huang, S., Bhagavatula, C., Ammanabrolu, P., Choi, Y., and Ren, X. 2023. SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Tasks. arXiv:2305.17390.
-
Zhou, A., Yan, K., Shlapentokh-Rothman, M., Wang, H., and Wang, Y.-X. 2023. Language Agent Tree Search Unifies Reasoning, Acting, and Planning in Language Models. arXiv:2310.04406.
-
Wang, M., Pang, A., Kan, Y., Pun, M.-O., Chen, C. S., and Huang, B. 2024. LLM-Assisted Light: Leveraging Large Language Model Capabilities for Human-Mimetic Traffic Signal Control in Complex Urban Environments. arXiv:2403.08337.
-
Guo, S., Deng, C., Wen, Y., Chen, H., Chang, Y., and Wang, J. 2024. DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoning. arXiv:2402.17453.
-
Wang, X., Zhou, K., Wu, W., Singh, H. S., Nan, F., Jin, S., Philip, A., Patnaik, S., Zhu, H., Singh, S., Prashant, P., Shen, Q., and Huang, B. 2025. Causal-Copilot: An Autonomous Causal Analysis Agent. arXiv:2504.13263.
-
Nottingham, K., Ammanabrolu, P., Suhr, A., Choi, Y., Hajishirzi, H., Singh, S., and Fox, R. 2023. Do Embodied Agents Dream of Pixelated Sheep?: Embodied Decision Making using Language Guided World Modelling. arXiv:2301.12050.
-
Gonzalez-Pumariega, G., Chen, W., Kedia, K., and Choudhury, S. 2024. Query-Efficient Planning with Language Models. arXiv:2412.06162.
-
Guo, J., Yang, B., Yoo, P., Lin, B. Y., Iwasawa, Y., and Matsuo, Y. 2023. Suspicion-Agent: Playing Imperfect Information Games with Theory of Mind Aware GPT-4. arXiv:2309.17277.
Response to Reviewer KPFr
Dear Reviewer KPFr, thank you again for your time and consideration. As the discussion period is nearing its end, we wanted to follow up on our response and the additional experiments. We have not yet heard back and would appreciate knowing whether our clarifications address your concerns, or whether any questions remain that we could help resolve. We would be happy to discuss them further. Thank you!
Response to Reviewer KPFr
Thank you for asking us to examine the paper’s central insights more closely. Your concerns about motivation and novelty prompted us to revisit the evidence for the attribution problem and explain what MPI adds to established experimental methods. Your suggestion to broaden the evaluation also led us to test additional decision algorithms.
1. Evidence from existing agent studies
Many agent studies compare their methods with other complete systems. In these comparisons, the policy can change along with the search or planning procedure (SwiftSage; LATS; Boomerang), available information (DS-Agent; Causal-Copilot; Suspicion-Agent), or tool support (LLM-Assisted Light). The performance difference therefore cannot be attributed to the policy alone. MPI asks a more specific question: how does the final outcome change when we replace the policy at one decision stage, keeping external information, candidate-construction rules, budget, and execution rules fixed?
Many of the comparisons above do not control decision support separately from the policy. This can lead researchers to credit the policy for gains that come from better support. A major source of this ambiguity that we address is action-space privilege: one policy receives curated actions, while another must construct or search for them. Even with the same raw observations, the candidate menu can contain the results of expert judgment or statistical computation performed before selection. The complete-system comparisons in DECKARD and LLM-Assisted Light, for example, do not separate action-interface and decision-support differences, respectively, from the contribution of the policy. MPI makes candidate construction and supporting computations explicit, then compares policies under matched conditions.
The gap is in connecting a component-level explanation to an experiment that tests that component. MPI provides a formal framework for defining the boundary between policy and support and measuring a specified policy’s contribution while retaining shared support.