- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper presents RunningTab which is a framework that provides an summary of workspace interactions for the agent. This includes task requirements, file reads, and listed but unopened files. RunningTab is then validated on three benchmarks with three LLMs, where it consistently outperforms plain DWI.
Critical strengths
- The paper is clearly written, with clear examples and it is very easy to follow.
- The experiments are cleanly designed and offer good benchmark and model variations.
Critical weaknesses
- The main contribution of this paper is the idea of keeping track of workspace/file interactions. For larger model sizes thinking tokens already do this. Even for the smaller models, it is not clear whether thinking tokens where used or not, or whether using a simple memory would have the same effect. Therefore the paper lacks novelty or informativeness.
Questions for authors
- Was the experiment done with thinking tokens on? If yes, what was the thinking level?
- If thinking tokens were on, was there any experiment done on whether the information in running tab or a subset exist in the thinking tokens?
- Does Pi agent keeps a memory? Would the results be different if the used an agent that keeps a memory?
- Why the experiment was not performed on larger models with more reasoning skills?
3 comments
Response to Reviewer ZxnH (2/2)
Q1. Was thinking on, and if so, at which level?
Yes, thinking was on in all runs: GPT-5.4 nano runs with high reasoning effort, and DeepSeek V4 Flash and Gemini 3.8 Flash with their default thinking.
Q2. Does the content of the tab, or a subset of it, appear in the thinking tokens?
Mostly not. Using the audit described in our response to Weakness 1.1, we find that of the values that the tab would hold but the agent did not include in its deliverable, only 30.8% appear in the full reasoning of DeepSeek V4 Flash.
Q3. Does Pi keep a memory, and would the results differ with an agent that keeps one?
Pi keeps no memory beyond the context window, and its context management is designed to truncate long observations and clear older ones as they accumulate (Section 4). As for an agent that keeps a memory, Model-Kept Notebook gives the agent a memory outside the context window that it writes itself, and RunningTab outperforms it with all three LLMs (Table C.1), as detailed in our response to Weakness 1.2.
Q4. Why were the experiments not performed on larger models with more reasoning skills?
Following your suggestion, we have additionally run experiments with two larger models of the GPT-5.4 family, GPT-5.4 mini and the flagship GPT-5.4, both with high reasoning effort as with GPT-5.4 nano. As shown in Table C.3, RunningTab outperforms DCI with all three models on both metrics, including the flagship GPT-5.4. This indicates that RunningTab remains beneficial regardless of model size: even with stronger reasoning, the agent still benefits from having the environment keep track of every file that it reads or sees listed.
Table C.3. DCI and RunningTab on Workspace-Bench across the GPT-5.4 family (Pass Rate / TCR@70, three-run mean).
| Method | GPT-5.4 nano | GPT-5.4 mini | GPT-5.4 |
|---|---|---|---|
| DCI | 41.63 / 20.67 | 44.03 / 25.33 | 49.53 / 30.33 |
| RunningTab | 44.73 / 25.67 | 46.28 / 27.67 | 51.09 / 33.00 |
Dear Reviewer ZxnH,
As the discussion period closes on November 18 (AoE), we would like to gently remind you of our responses, which show that thinking keeps few of the values that the agent misses, that RunningTab outperforms two memory baselines, and that its gains hold with the larger GPT-5.4 mini and GPT-5.4. We would be deeply grateful if you could take a moment to review them. Your questions led to several of our new analyses, including the audit of thinking and the experiments with larger LLMs, and we sincerely hope that they address your concerns.
Warm regards, Authors
Response to Reviewer ZxnH (1/2)
We sincerely thank you for your careful review. We are also grateful that you found the paper easy to follow with clear examples, and the experiments cleanly designed with good benchmark and model variations. We have carefully considered all of your comments and address each of them below with additional analyses and new experiments.
Thank you for raising this point. We would first like to clarify that thinking was on in all runs (high reasoning effort for GPT-5.4 nano and default thinking for DeepSeek V4 Flash and Gemini 3.8 Flash), but even so, we observe that thinking often does not keep track of the workspace interactions. To verify this, we have audited the reasoning of DeepSeek V4 Flash, the strongest of our three LLMs and the one that exposes its full reasoning. Specifically, for each value that the agent saw in the workspace, we checked whether the value appears in the reasoning and whether it appears in the deliverable. We then find that, while the reasoning mentions most of the values that the agent included in the deliverable (79.0%), the real problem lies in the values that the agent saw but did not include in the deliverable, only 28.2% of which appear in the reasoning. In other words, most of the values that the task requires but the agent misses do not appear in its reasoning, which means that thinking does not reliably keep a record of what the agent has seen. RunningTab, in contrast, records the files that the agent reads in the tab, independently of its reasoning, and brings them back to each open requirement, raising the Pass Rate of DeepSeek V4 Flash on Workspace-Bench from 53.28 to 59.99 (Table 1). This gain also holds for larger models with stronger reasoning: with GPT-5.4 mini and the flagship GPT-5.4, RunningTab still outperforms DCI, as detailed in our response to Question 4 (Table C.3).
Following your suggestion, we have compared RunningTab with two memory approaches, and RunningTab outperforms both. The first is Model-Kept Notebook, a memory that the agent keeps itself: it is identical to RunningTab, except that the agent has to record the files it reads and sees listed in the notebook itself, and it stays below RunningTab with all three LLMs (Table C.1). The second is Read Log Only, a memory of the file interactions that the environment records, without requirements or checks, and it also stays well below RunningTab (Table C.2). The advantage of RunningTab therefore comes from the link that neither memory has: the environment records every file (unlike the notebook), and the review links these files to each open requirement (unlike Read Log Only).
Most importantly, these results reveal the novelty of RunningTab, which lies not in keeping track of workspace interactions itself but in who records them: neither the reasoning of the agent (Weakness 1.1) nor a memory that the agent writes (Weakness 1.2) records them reliably, whereas RunningTab has the environment record every file that the agent reads or sees listed, and to the best of our knowledge, it is the first method that lets the environment keep this record.
Table C.1. Model-Kept Notebook on Workspace-Bench with the three LLMs (Pass Rate / TCR@70, three-run mean).
Table C.2. Read Log Only on Workspace-Bench with GPT-5.4 nano (three-run mean).