acceptodds
Under review as a conference paper at ICLR 2027

ACI-Core: How Software Semantics Shape Language Agents' Task Execution

Abstract

As large language models are increasingly used to operate application software, prior work has largely focused on tool invocation, agent-oriented interface design, and graphical user interfaces, while a deeper question remains underexplored: how do different types of software runtime information affect an agent's task execution? We introduce ACI-Core, which models a stateful application as a system of observable state transitions. Through an external and pluggable layer, ACI-Core turns software information, such as action semantics, execution conditions, and execution history, into controllable experimental variables without modifying the application's business logic. Across multiple controlled experiments, we find that multiple types of information, including execution history and structured goals, affect task completion and execution cost. In MailWorld, exposing a History bundle increased terminal Oracle success by 57.5 percentage points, while exposing a structured Goal increased success by 32.5 percentage points in a separate cohort. Full-event History further reached 95% terminal Oracle success, but used 3.33 times the input tokens of bounded History. The largest observed change in task completion approached 60 percentage points. In a delayed-evidence reuse experiment on τ-bench retail tasks, targeted history retrieval substantially improved first exact-binding reliability, matching full history on first exact binding while using 76.8% fewer input tokens. Our study of ACI-Core suggests that effective software operation by AI agents is shaped by what task-relevant information is available, when it becomes available, and how it is selected and preserved. All experimental code and data are available in the anonymous repository: [https://anonymous.4open.science/r/Your-anonymizations9527-8B71](https://anonymous.4open.science/r/Your-anonymizations9527-8B71).

Primary area:
foundation or frontier models, including LLMs
Keywords:
language model agents, agent-computer interfaces, software semantics, software interaction, tool use, context management

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_uK1POpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ACI-Core, a framework that models software execution as state transitions and enables controlled studies of language agents by selectively exposing different types of runtime information. By varying information disclosure while keeping the underlying runtime fixed, the authors investigate what information matters, how evidence should be retained or recovered, and when it should be exposed. Experiments show that History and Structured Goals improve task completion in MailWorld, while targeted history retrieval achieves effective delayed evidence reuse with substantially lower token costs than full-history replay.

Critical strengths
  1. Separating information disclosure from runtime enforcement

ACI-Core varies what agents can see while retaining backend checks, helping isolate the effect of information availability from that of execution safeguards.

  1. Controlling evidence retention and exposure timing

The framework supports comparisons of bounded, full, and targeted history, as well as different disclosure schedules, enabling systematic study of how and when agents should receive task-relevant evidence.

Critical weaknesses
  1. Lack of a concrete agent improvement method

ACI-Core provides a useful framework for analyzing how information exposure affects agent behavior, but the paper mainly focuses on measuring the value of different information channels. So the work dont directly demonstrate how these insights can be translated into improved agent performance.

  1. The ten-channel decomposition is insufficiently justified.

The channels form a practical engineering taxonomy, but the paper does not clearly explain why these ten categories are preferable to alternative decompositions or how their overlap is handled.

Questions for authors
  1. Do the interventions change only information visibility? Please provide representative model inputs and runtime differences for Full, History-off, and Subgoal-off. Why can Subgoal-off cause progress-update rejection? Do the main effects remain when reminders, action legality, and update behavior are held fixed?

  2. How does Targeted History select evidence?

The paper should clarify the implementation details of the retrieval mechanism, including query construction, ranking strategy, retrieval budgets, and when the policy was frozen. It's unclear whether retrieval operates purely on information available to the agent during execution or whether it relies on task annotations or evaluator-derived information. This distinction is critical for determining whether the improvement reported in Table 3 represents a realistic recovery mechanism that can be deployed in autonomous agents.

  1. Will the principal results robust to execution-selection rules?

Please clarify which main-table cells are affected by post-cap recoveries and tool-choice configuration changes. It would be helpful to provide sensitivity analyses that retain the original valid failures and strictly follow the original budget constraints. And please indicate whether terminal-completion results are available on independent tasks evaluated after the policies were frozen.

7 comments
AuthorsOpenReview ↗

Response to uK1P: from measurement to improved performance, and why ten channels

We thank the reviewer. Section and appendix numbers refer to the revised PDF. We stand ready and look forward to further discussion with you. New experiments were frozen before any call; a:b counts tasks solved only under the first / only under the second condition.

W1. From measurement to improved performance

Research goal. ACI-Core is a research study, not an application or new agent method. Its goal is to understand how software information affects agents on stateful tasks, and to find ways of providing it that balance quality and cost. The findings translate directly into gains with the same model and budget:

  • Structure the execution history (App. C.4). DeepSeek, same tasks, Oracle and budgets. Against native function calling that keeps its complete assistant/tool transcript, ACI with full history scores 16/16 vs 8/16 and 15/16 vs 9/16 in two rounds (8:0, 6:0). Over 12 tasks observed in all four trials: 23/24 vs 16/24 (+29 pp); both rounds favor ACI, no task favors the baseline, and ACI needs under half the decisions (11.7 vs 26.2). Against Raw with full HTTP history: 19/20 vs 8/20 (11:0).
  • Trade quality against cost. Bounded ACI history beats Raw (13/20 vs 8/20) at half the input.
  • Recover delayed evidence. Targeted recovery raises first exact binding from 2/8 to 8/8 and final reward from 2/8 to 5/8 over bounded history, with 76.8% fewer input tokens than full history.
  • Restate the goal. For DeepSeek, the verbatim Goal scores 11/20 vs 4/20 for length-matched generic text (7:0).
  • Disclose before the decision. Correct first bindings: 9/10 with evidence always visible, 0/10 when late.

W2. Why ten channels?

The design is inspired by state-transition systems, I/O Automata, Statecharts, program logic and planning (§2), which supply the concepts of actions, conditions, states, observations, goals and task decomposition. Building on them, we turn the information in stateful software tasks into ten channels, each shown or hidden while Runtime and Oracle stay fixed, grouped as static contract, task state and execution information. It is a practical decomposition, not claimed unique or optimal. Channels may overlap, so each reported effect is one channel's increment with all others visible (Eq. 3); the Goal placebo (C.3) and History split (C.2) test the main overlaps. Security/permission semantics and failure recovery are left to future extensions.

Q1. Do interventions change only visibility?

Inputs. First-turn inputs for Full, History-off and Subgoal-off are in the repository; App. C.1 lists every change. Goal, Action Semantics, Observations and Identity change visibility only; hidden Conditions or Types stay enforced with generic rejections; hidden Dynamic Update hides state-change events; History-off removes records, the history-read tool, reminders and their instructions.

Subgoal-off. Progress updates are accepted and the hidden Subgoal tracked; only a replacement that skips review of the previous Subgoal is rejected, and business actions continue. We corrected §3.1, which said hiding never changes acceptance.

Holding factors fixed. Action legality is identical in all arms; the Goal ablation changes no Runtime behavior. The History advantage holds with identical instructions (GPT 8:0, DeepSeek 6:0; App. C.2) and with reminders off in both arms (LifeWorld 0/15 → 6/15).

Q2. Targeted History

  • Candidates and ranking: successful Action records the agent has already seen, ranked by keyword overlap with the user instruction (weight 3), current Subgoal (4), recent feedback (2), and current Phase and available Actions (1 each).
  • Budget: at most 3 records, 8,192 bytes, minimum score 4; skipped if a newer version is visible.
  • Freeze: fixed in one code version before the runs; no per-task tuning.
  • Information used: only what an autonomous agent sees at run time; no annotations or evaluator output.

Q3. Robustness to selection rules

  • Budget-cap recovery: two recovered runs change two GPT cells by one task each (Full 18/19 → 19/20; D39 17/19 → 18/20); no conclusion changes.
  • Tool choice: named in all MailWorld cohorts; required only in 56 GPT replacements on authored backends (55 failed).
  • Valid failures: never replaced; only infrastructure-invalid attempts are, once, and just one of 65 replacements succeeded.
  • Independent post-freeze results: yes. On 30 unused official τ-retail test tasks, chosen by a fixed rule before any call, the frozen configuration has no protocol failure and reaches reward 1 on 20/30 (App. C.5).

Presentation and code

Figure 2 and page 5 are fixed. The revised paper corrects the repository link formatting; the abstract's link, https://anonymous.4open.science/r/Your-anonymizations9527-8B71 , opens normally, and the code and results of all new experiments are now available there.

AuthorsOpenReview ↗

Response to uK1P: We have added the following experiments

While awaiting your further assessment, we have conducted the following additional experiments in the hope of addressing more of your questions.

Q1 (Does hiding Subgoals alter update acceptance, and does the effect remain when acceptance is controlled?) We independently varied Subgoal visibility and the acceptance rule. Under strict acceptance, visible Subgoals had a 35.0-percentage-point advantage (20 paired tasks; 95% task-level bootstrap interval: 10.0 to 60.0). The advantage remained 31.6 percentage points under permissive acceptance (19 paired tasks; interval: 10.5 to 52.6).

Thank you for the clarifications and additional experiments. Some aspects of the correspondence between the paper’s core methods and the released implementation remain unclear to me. In the artifact package I inspected, code/rebuttal/diff_first_turn_inputs.py imports aci_core.driver, E2AgentDriver, and experiment-condition construction modules whose source I could not find in the package. These dependencies appear central to constructing the information interventions and per-decision context. I also could not locate the implementation of Targeted History’s candidate filtering, weighted ranking, deduplication, and context injection. The written clarification is helpful, but the corresponding implementation would be needed to check these details against the reported experiments. This leaves some uncertainty about how the described methodology maps to the code used to obtain the main results.

Reviewer_43wDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ACI-Core, a framework for systematically studying how model-visible software information affects language agents' task execution. It separates information disclosure from runtime enforcement and organizes software information into ten controllable channels. Experiments across multiple software environments investigate which information matters, how evidence should be retained, and when it should be exposed. The results show substantial benefits from History and Structured Goal in MailWorld, while targeted history retrieval improves exact evidence binding with lower token consumption. However, the effects vary across environments.

Critical strengths
  1. ACI-Core provides a useful experimental framework that isolates the effects of software information disclosure while keeping runtime enforcement and evaluation fixed. This enables more systematic investigation of agent behavior than conventional task-completion benchmarks.
  2. The experiments provide meaningful insights into information availability and efficiency. Table 1 shows large improvements from History (+57.5 pp) and Structured Goal (+32.5 pp), while Table 3 demonstrates that targeted retrieval achieves the same observed first-binding success as Full History with 76.8% fewer input tokens.
Critical weaknesses
  1. Limited generalization. Results are mainly concentrated in MailWorld, while cross-software experiments use few tasks and show severe floor effects. Native τ-bench evaluations also reveal compatibility issues.
  2. Incomplete isolation of effects. Some ablations modify multiple information components simultaneously. Targeted retrieval is also insufficiently specified and evaluated on only four training tasks.
  3. Minor formatting issue. Page 5 has excessive white space and could benefit from a more balanced layout.
Questions for authors
  1. Could the authors clarify how Targeted History selects relevant records and whether task-specific tuning is involved?
  2. Have the native τ-bench compatibility issues been resolved and validated on an independent task set?
5 comments
AuthorsOpenReview ↗

Response to 43wD: generalization scope, effect isolation, and native τ validation

We thank the reviewer for the constructive assessment. Section and appendix numbers refer to the revised PDF. We stand ready and look forward to further discussion with you. New experiments were frozen before any call; a:b counts tasks solved only under the first / only under the second condition.

W1. Generalization

ACI-Core's goal is to understand how software semantics and execution information affect agents on stateful tasks, and which ways of providing information balance quality and cost. MailWorld supports the full ten-channel analysis; the other settings test which findings transfer, rather than assume uniform gains.

  • Independent official tasks. 30 previously unused τ-retail test tasks, scored by the official evaluator, validate the repaired native protocol (Q2).
  • Cross-software transfer. Exact Evidence lowers mean input tokens in all four settings (by 1.7k to 160.8k per task pair). History raises completion in BlogWorld, LifeWorld and the τ adaptation (+2, +6 and +1 of 15), and Goal in LifeWorld and τ (+3 each).
  • Across models and repeats. In MailWorld, History helps both model families and reproduces in a new frozen panel (8:0, 6:0); DeepSeek's Goal effect reproduces against a placebo (7:0); and ACI with full history beats a native function-calling agent that keeps its full transcript in two independent rounds (16/16 vs 8/16, 15/16 vs 9/16; App. C.4).

W2. Isolation of effects

History (App. C.2). H0 keeps records, the history-read tool and record-dependent reminders; H1 removes them while keeping the generic instructions identical; H2 also removes those instructions. H0 vs H1: GPT 8:0, DeepSeek 6:0; H1 vs H2: no detectable difference. The four-software History contrast keeps reminders off in both arms, and LifeWorld still rises from 0/15 to 6/15. The effect therefore comes from records and read access rather than the deleted instructions, and it persists without reminders.

Structured Goal (App. C.3). The field repeats the visible instruction verbatim and changes no Runtime behavior. Against no field (K0) and length-matched generic text (K2), the verbatim field (K1) gives DeepSeek 11/20 vs 4/20 for K2 (7:0; +35 pp, bootstrap 95% CI [15, 55]), and K2 does not beat K0 (0:1). Task content, not added length, carries the effect.

All channels (App. C.1). A new table lists, for each ablation, exactly which fields, instructions and Runtime behaviors change. Structured Goal, Action Semantics, Observations and Identity change visibility only.

Targeted retrieval. It is now fully specified (Q1; §4.3.2, App. D.1). The study is designed to isolate recovery: a shared prefix fixes what the agent has acquired, so only the History policy differs. On four official training tasks, it raises first exact binding from 2/8 to 8/8 and final reward from 2/8 to 5/8 relative to bounded history, with 58.8% fewer input tokens than bounded and 76.8% fewer than full history.

Q1. How Targeted History selects records

  • Candidates: only successful Action records the agent has already seen.
  • Scoring: keyword overlap with the user instruction (weight 3), the current Subgoal (4), recent feedback (2), and the current Phase and available Actions (1 each).
  • Budget: at most 3 records within 8,192 bytes, minimum score 4, without duplicates; a record is skipped if a newer version of it is already visible.
  • Inputs: the user instruction and the runtime view only; no task annotations, Oracle or evaluator output.
  • No task-specific tuning: the policy and all conditions were fixed in one code version before the runs, with no per-task parameters.

Q2. Native τ compatibility and independent validation

Yes on both counts. A/B/C are three ACI Contract variants (App. G). The original failures were protocol failures (73 runs without an official reward): 26 illegal top-level responses, 21 budget terminations and 26 premature finishes.

PanelRaw reward 1ACI reward 1 (A/B/C)
Original 30 tasks, before repair22/303/0/1
Same 30 tasks, after repair22/3021/24/16

Independent validation (App. C.5). On 30 unused official test tasks, selected by a rule fixed before any call and sharing neither tasks nor users with the repair panel, the frozen configuration B has no protocol failure and reaches reward 1 on 20/30 under the official evaluator. The repair therefore carries over to independent tasks.

W3. Presentation and reproducibility

Page 5 is fixed: tables now float, and the main text still fits in nine pages; Figure 2 is also fixed. The revised paper corrects the anonymous-repository link formatting. The abstract's link, https://anonymous.4open.science/r/Your-anonymizations9527-8B71 , opens normally, and the code and results of all new experiments are now available there.

AuthorsOpenReview ↗

Response to 43wD: We have added the following experiments

While awaiting your further assessment, we have conducted the following additional experiments in the hope of addressing more of your questions.

Q2 (Have the native τ-bench compatibility issues been resolved and validated on independent tasks?) We evaluated the frozen ACI configuration B on 30 official test tasks that were not used to repair the protocol. The configuration produced classifiable terminal outcomes on all 30 tasks, with no terminal protocol failures, and received official reward 1 on 20 tasks.

AuthorsOpenReview ↗

Response to 43wD

While awaiting your further feedback, we have conducted the following additional experiments, which we hope will address more of your questions.

Thank you for recognizing the value of the ACI-Core experimental framework and for your constructive suggestions on effect isolation and cross-software evaluation. Using a predefined information-channel taxonomy, our study examines which information helps under which task and software conditions. Building on our previous replies, we have added the following experiments.

1. We completed an independently controlled historical-detail experiment. Both arms retained identical recording, history-read tools, immediate feedback, and update-acceptance rules, with reminders disabled in both; only the visibility of historical business details changed. On 20 MailWorld tasks with two runs each, DeepSeek Pro achieved 25/40 with details versus 2/40 with indices alone: +57.5 percentage points (95% task-paired bootstrap interval, 37.5 to 75.0), with 47.4% fewer cumulative input tokens. The two runs yielded 13/20 versus 1/20 and 12/20 versus 1/20. This addresses your W2 concern that some ablations change several components simultaneously and therefore incompletely isolate their effects.

2. We completed autonomous retrieval without a scripted acquisition prefix. On a direct-instruction adaptation of 20 independently assigned official tau-retail tasks, DeepSeek Flash achieved 20/20 with Targeted History, 17/20 with bounded history, 17/20 with ordinary lexical retrieval, and 20/20 with full history. The observed Targeted-minus-bounded difference was +15.0 points (95% task-paired interval, 0.0 to 30.0); Targeted used 5.5% fewer cumulative input tokens than full history. The policy was fixed before evaluation and used only previously visible records and current visible context, without per-task tuning or Oracle information. This addresses your Q1 on task-specific tuning and W2 on the small retrieval evaluation, adding autonomous terminal-completion results.

3. We completed a configuration-transfer experiment on Gitea, a real public software system. On 20 author-created tasks, the ACI default configuration, fixed before evaluation, achieved 20/20 with DeepSeek Flash; the conventional full-transcript function-calling baseline also achieved 20/20. The tasks cover ten workflow families, including ten medium and ten hard tasks. This addresses your W1 concern about concentration on MailWorld and limited cross-software evaluation, providing further evidence of implementability after software-specific adaptation.

Thank you for these helpful suggestions. We hope the additional results clarify the independent contribution of information content and the operability of the policies in autonomous tasks and new software.

Reviewer_4vZMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper proposes ACI-Core, a layer between an LLM agent and a stateful application. It splits what the agent sees into ten channels (Types, Action Semantics, Conditions, Phase, Identity, Structured Goal, Subgoal, History, Observations, Dynamic Update). Any channel can be hidden while the runtime still enforces the rules and an independent Oracle checks the final state. In MailWorld ablations with two models, hiding History drops success by 57.5 pp and hiding Structured Goal drops it by 32.5 pp. Hiding Action Semantics, Conditions or Dynamic Update slightly improves success. Smaller studies on three more environments and a τ-bench adaptation report partial transfer, token savings from Exact Evidence, and a targeted history method that matches full history on first correct binding with fewer tokens.

Critical strengths

Separating what the agent sees from what the runtime enforces is a clean idea. Together with strict Oracle grading, it makes "which information helps agents" a controlled question.

The reporting is honest. Null and negative results are included, and failures and reruns are documented in the appendix.

Critical weaknesses

The findings do not support the "software semantics" framing, and the two big effects are confounded. The semantic channels (Action Semantics, Conditions, Dynamic Update) show no benefit. The large effects come from History and Goal, which are memory and task restatement. Turning History off leaves the agent with no record of its own actions and also disables reminders, so agents just run out of budget (Table 7). No normal agent works like that. The more useful comparison is bounded vs full history, where all 12 extra successes come from DeepSeek (7 to 19) and GPT gains nothing (19 to 19) for 3.33x the tokens. Structured Goal repeats the task while the original instruction is still visible, and there is no placebo of similar length. Its construction is also not described. Subgoal off can cause update rejection, which contradicts the claim that hiding a channel never changes what the driver accepts.

There is also no evidence that ACI beats a plain interface. DeepSeek scores 7/8 at the minimal S0 level (Fig. 4). On native τ-bench, Raw gets reward 1 on 22/30 tasks while ACI configs get 3, 0 and 1 (Table 15). This is not mentioned in the main text, and no comparison with a standard function calling agent is given.

The experiments are too small and noisy for the claims made. Each cell is a single run with no seed. The two Full controls differ by 3 to 4 successes out of 20, which is as large as most reported effects. A 25 pp DeepSeek change for Dynamic Update is treated as noise, but a 25 pp GPT change for Goal is treated as a finding. The other studies are pilots: 5 tasks per environment in the transfer panel, n=8 for Phase and targeted history, and 10 episodes with one model for timing. The only held out test (Table 7) gives 0/4 success in both arms. All tasks and Oracles are written by the authors, many are selected to favor the treatment, and results rely on author approved recovery views and 65 replacement runs.

Smaller issues: undefined labels (D39, V12, P3B, F0R0, Modcon), a sign convention that changes across tables, a placeholder repo link and other draft leftovers, and Figures 1 and 2 are too small to read.

Questions for authors

How is Structured Goal generated, and does it include anything beyond the user instruction? What happens with a placebo of similar length?

How does the best ACI setup compare with a plain function calling agent that keeps its full transcript, on the same tasks?

Do Cohorts A and B use the same tasks? Can you repeat the main cells and report intervals?

How does Targeted History rank records, and does it use any task specific or Oracle knowledge?

Why does ACI fail on native τ-bench, and how does the fixed version do on all 30 tasks?

Who decides what goes into an Evidence card?

5 comments
AuthorsOpenReview ↗

Response to 4vZM: revised framing, new History isolation panel, native function-calling baseline

We thank the reviewer. Section and appendix numbers refer to the revised PDF. We stand ready and look forward to further discussion with you. New experiments were frozen before any call; a:b counts tasks solved only under the first / only under the second condition.

1. Framing and the two large effects

Research goal. ACI-Core studies how software information affects agents on stateful tasks, and which ways of providing it balance quality and cost. Ablations such as History-off measure a channel's value; they are not agent designs (realistic agents: Part 2). The revised framing covers both kinds of information: under an enforcing Runtime, execution history is worth far more than action descriptions, while Conditions help on tasks that need them (5/9 vs 3/9; avoidable precondition failures 5 vs 19; App. C.7).

History. A new panel (App. C.2) separates the bundle: H0 keeps records, the history-read tool and reminders; H1 removes them but keeps identical generic instructions; H2 also removes those instructions. H0 vs H1: GPT 8:0, DeepSeek 6:0; H1 vs H2: no detectable difference. The gap comes from records and their reminders, not from deleted instructions. With reminders off in both arms, LifeWorld still rises 0/15 → 6/15.

Bounded vs full. Full history's value is model-dependent: it lifts DeepSeek from 7/20 to 19/20, while GPT already reaches 19/20 with bounded history. A new concurrent DeepSeek panel (App. C.4) reproduces the gain (19/20 vs 13/20, 7:1).

Structured Goal (Q1). total_goal is a verbatim copy of the user instruction and adds nothing else (App. C.1). Against length-matched generic text (placebo, App. C.3), DeepSeek's verbatim field scores 11/20 vs 4/20 (7:0; +35 pp, 95% CI [15, 55]), and the placebo does not beat no field (0:1). Task content, not added length, carries the effect.

Subgoal. The incorrect sentence is removed: with Subgoal hidden, only a replacement skipping review of the previous Subgoal is rejected; all other actions are accepted.

2. ACI vs a plain interface (Q2, Q5)

Q2: native function calling with full transcript (new, App. C.4). Same 20 MailWorld tasks, model (DeepSeek), Oracle and budgets; two rounds. ACI full history vs a function-calling agent keeping its complete assistant/tool transcript: 16/16 vs 8/16 and 15/16 vs 9/16 on evaluable pairs (8:0, 6:0). Over 12 tasks observed in all four trials: 23/24 vs 16/24 (+29 pp). Both rounds favor ACI, no task favors the baseline, and ACI needs fewer than half the decisions (11.7 vs 26.2).

Raw with full HTTP history. 8/20 vs ACI full history 19/20 (11:0; +55 pp, 95% CI [35, 75]); bounded ACI already beats Raw (13/20) at half Raw's input.

Fig. 4. "S0" is the native Raw API, now labeled L0; both models reach 8/8 at the full rungs L5–L6.

Q5: native τ. The original failures were protocol failures (73 runs without an official reward: 26 illegal top-level responses, 21 budget terminations, 26 premature finishes), now in the main text. After repair, on the same 30 tasks ACI variants A/B/C (App. G) reach reward 1 on 21/24/16 vs Raw 22. On 30 unused official test tasks (App. C.5), none of these failures recurs, and B reaches reward 1 on 20/30.

3. Size, repeats and selection (Q3)

  • Same tasks. Cohorts A and B use the same 20 tasks (identical hashes).
  • Repeats and one standard. An effect is claimed only when its paired test is significant and it reproduces in a new frozen panel, so both 25 pp changes cited are treated alike. History (16 and 7 of 20 tasks; new panel 8:0, 6:0) and DeepSeek's Goal (8 of 20; placebo 7:0) pass and exceed the 3–4-task gap between Full controls. Intervals: App. C.3–C.4.
  • Selection. The main cohorts use the whole 20-task super-complex tier, not tasks chosen for any treatment; the τ panels use official tasks and evaluator. Two runs recovered after the budget cap change two cells by one task each, and only one of the 65 replacement outcomes is a success.

4. Retrieval, cards and presentation (Q4, Q6)

Q4. Previously seen successful records are ranked by keyword overlap with the instruction (weight 3), current Subgoal (4), recent feedback (2), Phase and available Actions (1 each); at most 3 records, 8,192 bytes, minimum score 4. No Oracle, annotations or per-task parameters; fixed in one code version before the runs.

Q6. The experimenters fix cards in advance: fields the agent already observed (MailWorld, authored software), a field snapshot (τ adaptation), or a fact card identical across timing arms.

Presentation and code. Labels are defined or renamed; Table 1 uses Full − ablated (Eq. 3); Figures 1–2, page 5 and leftovers are fixed. The revised paper corrects the repository link formatting. The abstract's link, https://anonymous.4open.science/r/Your-anonymizations9527-8B71 , is not a placeholder and opens normally; code and results of all new experiments are now available there.

AuthorsOpenReview ↗

Response to 4vZM: We have added the following experiments

While awaiting your further assessment, we have conducted the following additional experiments in the hope of addressing more of your questions.

Q3 (Could you repeat the main experiments and report intervals?) We repeated the History comparison across multiple runs and computed task-paired intervals. Full exceeded minus-History by 58.3 percentage points for DeepSeek V4 Pro (16 complete paired tasks; 95% task-level bootstrap interval: 39.6 to 77.1) and by 60.5 percentage points for GPT-5.6 Sol (19 paired tasks; interval: 44.7 to 76.3). The difference was in the same direction in every run for both models.

Regarding the concern that Subgoal updates can be rejected: We independently varied Subgoal visibility and the acceptance rule. Under strict acceptance, visible Subgoals had a 35.0-percentage-point advantage (20 paired tasks; interval: 10.0 to 60.0). The advantage remained 31.6 percentage points under permissive acceptance (19 paired tasks; interval: 10.5 to 52.6).

AuthorsOpenReview ↗

Response to 4vZM

While awaiting your further feedback, we have conducted the following additional experiments, which we hope will address more of your questions.

Thank you for your careful assessment of software-information value and experimental reliability. Your comments have also helped us clarify our scope: ACI-Core provides a controlled framework for studying how the content, organization, retention, and disclosure timing of software information affect task execution, and under what conditions. The ten information channels follow a predefined taxonomy; we do not assume that every channel should improve success. Conventional-agent baselines help assess the practical relevance of the findings. The following evidence was obtained after our previous replies.

1. We completed a controlled experiment on API names and functionality descriptions. We used ten medium- and ten highest-difficulty MailWorld tasks, with two runs per task. Without functionality descriptions, meaningful versus anonymous names yielded 36/40 versus 18/40 successes for DeepSeek Flash and 37/40 versus 15/40 for GPT-5.6 Terra. Adding correct descriptions to the same anonymous names increased successes from 18/40 to 35/40 and from 15/40 to 35/40: +42.5 percentage points (95% task-paired bootstrap interval, 22.5 to 62.5) and +50.0 points (30.0 to 70.0). All arms used ordinary native function calling with full transcripts and identical operation capabilities, backend, evaluator, and budgets. Name anonymization serves as an information intervention to separate the contributions of name cues and explicit descriptions. This addresses your W1 concern about the value of software-semantic information and its conditions of usefulness.

2. We completed a historical-detail experiment with additional mechanism controls. Both arms retained identical recording, history-read tools, immediate feedback, and update-acceptance rules, with reminders disabled in both; only the visibility of historical business details changed. On 20 tasks with two runs each, DeepSeek Pro achieved 25/40 with details versus 2/40 with indices alone: +57.5 points (95% task-paired interval, 37.5 to 75.0), with 47.4% fewer cumulative input tokens. The two runs yielded 13/20 versus 1/20 and 12/20 versus 1/20. This addresses your W1 concern about reminder/read-mechanism confounding and W3 concern about further validation.

3. We completed autonomous retrieval without a scripted acquisition prefix. On a direct-instruction adaptation of 20 independently assigned official tau-retail tasks, DeepSeek Flash achieved 20/20 with Targeted History, 17/20 with bounded history, 17/20 with ordinary lexical retrieval, and 20/20 with full history. The observed Targeted-minus-bounded difference was +15.0 points (95% task-paired interval, 0.0 to 30.0); Targeted used 5.5% fewer cumulative input tokens than full history. The policy used only previously visible records and current visible context, without per-task tuning or Oracle information. This adds autonomous terminal evidence relevant to your Q4 on retrieval inputs and W3 on evaluation size.

4. We completed a fixed-configuration evaluation on Gitea, a real public software system. On 20 author-created tasks, both the ACI default configuration and a conventional full-transcript function-calling baseline achieved 20/20 with DeepSeek Flash. The tasks cover ten workflow families, with ten medium and ten hard tasks. This addresses your W3 concern about limited cross-software task coverage and adds a new-software reference to the Q2 baseline discussion. It provides evidence that the framework can execute tasks after software-specific adaptation.

Thank you for considering these additional results. We hope they help assess the independent effects, conditions of usefulness, and cross-software implementability of information interventions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.