- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ACI-Core, a framework that models software execution as state transitions and enables controlled studies of language agents by selectively exposing different types of runtime information. By varying information disclosure while keeping the underlying runtime fixed, the authors investigate what information matters, how evidence should be retained or recovered, and when it should be exposed. Experiments show that History and Structured Goals improve task completion in MailWorld, while targeted history retrieval achieves effective delayed evidence reuse with substantially lower token costs than full-history replay.
Critical strengths
- Separating information disclosure from runtime enforcement
ACI-Core varies what agents can see while retaining backend checks, helping isolate the effect of information availability from that of execution safeguards.
- Controlling evidence retention and exposure timing
The framework supports comparisons of bounded, full, and targeted history, as well as different disclosure schedules, enabling systematic study of how and when agents should receive task-relevant evidence.
Critical weaknesses
- Lack of a concrete agent improvement method
ACI-Core provides a useful framework for analyzing how information exposure affects agent behavior, but the paper mainly focuses on measuring the value of different information channels. So the work dont directly demonstrate how these insights can be translated into improved agent performance.
- The ten-channel decomposition is insufficiently justified.
The channels form a practical engineering taxonomy, but the paper does not clearly explain why these ten categories are preferable to alternative decompositions or how their overlap is handled.
Questions for authors
-
Do the interventions change only information visibility? Please provide representative model inputs and runtime differences for Full, History-off, and Subgoal-off. Why can Subgoal-off cause progress-update rejection? Do the main effects remain when reminders, action legality, and update behavior are held fixed?
-
How does Targeted History select evidence?
The paper should clarify the implementation details of the retrieval mechanism, including query construction, ranking strategy, retrieval budgets, and when the policy was frozen. It's unclear whether retrieval operates purely on information available to the agent during execution or whether it relies on task annotations or evaluator-derived information. This distinction is critical for determining whether the improvement reported in Table 3 represents a realistic recovery mechanism that can be deployed in autonomous agents.
- Will the principal results robust to execution-selection rules?
Please clarify which main-table cells are affected by post-cap recoveries and tool-choice configuration changes. It would be helpful to provide sensitivity analyses that retain the original valid failures and strictly follow the original budget constraints. And please indicate whether terminal-completion results are available on independent tasks evaluated after the policies were frozen.
7 comments
Response to uK1P: We have added the following experiments
While awaiting your further assessment, we have conducted the following additional experiments in the hope of addressing more of your questions.
Q1 (Does hiding Subgoals alter update acceptance, and does the effect remain when acceptance is controlled?) We independently varied Subgoal visibility and the acceptance rule. Under strict acceptance, visible Subgoals had a 35.0-percentage-point advantage (20 paired tasks; 95% task-level bootstrap interval: 10.0 to 60.0). The advantage remained 31.6 percentage points under permissive acceptance (19 paired tasks; interval: 10.5 to 52.6).
Thank you for the clarifications and additional experiments. Some aspects of the correspondence between the paper’s core methods and the released implementation remain unclear to me. In the artifact package I inspected, code/rebuttal/diff_first_turn_inputs.py imports aci_core.driver, E2AgentDriver, and experiment-condition construction modules whose source I could not find in the package. These dependencies appear central to constructing the information interventions and per-decision context. I also could not locate the implementation of Targeted History’s candidate filtering, weighted ranking, deduplication, and context injection. The written clarification is helpful, but the corresponding implementation would be needed to check these details against the reported experiments. This leaves some uncertainty about how the described methodology maps to the code used to obtain the main results.
Response to uK1P: from measurement to improved performance, and why ten channels
We thank the reviewer. Section and appendix numbers refer to the revised PDF. We stand ready and look forward to further discussion with you. New experiments were frozen before any call; a:b counts tasks solved only under the first / only under the second condition.
W1. From measurement to improved performance
Research goal. ACI-Core is a research study, not an application or new agent method. Its goal is to understand how software information affects agents on stateful tasks, and to find ways of providing it that balance quality and cost. The findings translate directly into gains with the same model and budget:
W2. Why ten channels?
The design is inspired by state-transition systems, I/O Automata, Statecharts, program logic and planning (§2), which supply the concepts of actions, conditions, states, observations, goals and task decomposition. Building on them, we turn the information in stateful software tasks into ten channels, each shown or hidden while Runtime and Oracle stay fixed, grouped as static contract, task state and execution information. It is a practical decomposition, not claimed unique or optimal. Channels may overlap, so each reported effect is one channel's increment with all others visible (Eq. 3); the Goal placebo (C.3) and History split (C.2) test the main overlaps. Security/permission semantics and failure recovery are left to future extensions.
Q1. Do interventions change only visibility?
Inputs. First-turn inputs for Full, History-off and Subgoal-off are in the repository; App. C.1 lists every change. Goal, Action Semantics, Observations and Identity change visibility only; hidden Conditions or Types stay enforced with generic rejections; hidden Dynamic Update hides state-change events; History-off removes records, the history-read tool, reminders and their instructions.
Subgoal-off. Progress updates are accepted and the hidden Subgoal tracked; only a replacement that skips review of the previous Subgoal is rejected, and business actions continue. We corrected §3.1, which said hiding never changes acceptance.
Holding factors fixed. Action legality is identical in all arms; the Goal ablation changes no Runtime behavior. The History advantage holds with identical instructions (GPT 8:0, DeepSeek 6:0; App. C.2) and with reminders off in both arms (LifeWorld 0/15 → 6/15).
Q2. Targeted History
Q3. Robustness to selection rules
Presentation and code
Figure 2 and page 5 are fixed. The revised paper corrects the repository link formatting; the abstract's link, https://anonymous.4open.science/r/Your-anonymizations9527-8B71 , opens normally, and the code and results of all new experiments are now available there.