acceptodds
Under review as a conference paper at ICLR 2027

RunningTab: Direct Workspace Interaction with Environment-Side Tabs

Abstract

Much knowledge work produces new deliverables from files a workspace already holds, and LLM agents are beginning to take such work over. Through direct corpus interaction, an agent can search and read any of those files from a terminal with no indexing, and producing a deliverable from many of them in this way is what we call direct workspace interaction (DWI). Reaching the files, however, is only half the task: nothing keeps track of what the task asks for, what has been read, and what was listed but never opened, all of which slip through the context window without leaving a trace, so an agent may extract a figure and still deliver a report without it. To address this, we present RunningTab, a framework that equips direct workspace interaction with an environment-side tab: a per-task record of what the task still owes, kept by the environment alongside the agent. Specifically, the agent adds its requirements, while the environment records every file read as an excerpt with its provenance and every listed but unopened file as a candidate; the agent can then see each requirement beside its best-matching excerpts and top unopened candidates, resolve it against matching content or set it aside with a reason, and, should it try to finish with requirements still open, receive them in a finish check. We validate RunningTab on three benchmarks with three LLMs, where it consistently outperforms plain DWI and baselines that keep the record in the model, while its tab usually holds the values a deliverable needs once seen.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
LLM Agents, Direct Workspace Interaction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZxnHOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents RunningTab which is a framework that provides an summary of workspace interactions for the agent. This includes task requirements, file reads, and listed but unopened files. RunningTab is then validated on three benchmarks with three LLMs, where it consistently outperforms plain DWI.

Critical strengths
  • The paper is clearly written, with clear examples and it is very easy to follow.
  • The experiments are cleanly designed and offer good benchmark and model variations.
Critical weaknesses
  • The main contribution of this paper is the idea of keeping track of workspace/file interactions. For larger model sizes thinking tokens already do this. Even for the smaller models, it is not clear whether thinking tokens where used or not, or whether using a simple memory would have the same effect. Therefore the paper lacks novelty or informativeness.
Questions for authors
  • Was the experiment done with thinking tokens on? If yes, what was the thinking level?
  • If thinking tokens were on, was there any experiment done on whether the information in running tab or a subset exist in the thinking tokens?
  • Does Pi agent keeps a memory? Would the results be different if the used an agent that keeps a memory?
  • Why the experiment was not performed on larger models with more reasoning skills?
3 comments
AuthorsOpenReview ↗

Response to Reviewer ZxnH (1/2)

We sincerely thank you for your careful review. We are also grateful that you found the paper easy to follow with clear examples, and the experiments cleanly designed with good benchmark and model variations. We have carefully considered all of your comments and address each of them below with additional analyses and new experiments.


W1.1. For larger models, thinking tokens may already keep track of workspace interactions, and for smaller models, it is unclear whether thinking was used.

Thank you for raising this point. We would first like to clarify that thinking was on in all runs (high reasoning effort for GPT-5.4 nano and default thinking for DeepSeek V4 Flash and Gemini 3.8 Flash), but even so, we observe that thinking often does not keep track of the workspace interactions. To verify this, we have audited the reasoning of DeepSeek V4 Flash, the strongest of our three LLMs and the one that exposes its full reasoning. Specifically, for each value that the agent saw in the workspace, we checked whether the value appears in the reasoning and whether it appears in the deliverable. We then find that, while the reasoning mentions most of the values that the agent included in the deliverable (79.0%), the real problem lies in the values that the agent saw but did not include in the deliverable, only 28.2% of which appear in the reasoning. In other words, most of the values that the task requires but the agent misses do not appear in its reasoning, which means that thinking does not reliably keep a record of what the agent has seen. RunningTab, in contrast, records the files that the agent reads in the tab, independently of its reasoning, and brings them back to each open requirement, raising the Pass Rate of DeepSeek V4 Flash on Workspace-Bench from 53.28 to 59.99 (Table 1). This gain also holds for larger models with stronger reasoning: with GPT-5.4 mini and the flagship GPT-5.4, RunningTab still outperforms DCI, as detailed in our response to Question 4 (Table C.3).


W1.2. A simple memory may have the same effect.

Following your suggestion, we have compared RunningTab with two memory approaches, and RunningTab outperforms both. The first is Model-Kept Notebook, a memory that the agent keeps itself: it is identical to RunningTab, except that the agent has to record the files it reads and sees listed in the notebook itself, and it stays below RunningTab with all three LLMs (Table C.1). The second is Read Log Only, a memory of the file interactions that the environment records, without requirements or checks, and it also stays well below RunningTab (Table C.2). The advantage of RunningTab therefore comes from the link that neither memory has: the environment records every file (unlike the notebook), and the review links these files to each open requirement (unlike Read Log Only).

Most importantly, these results reveal the novelty of RunningTab, which lies not in keeping track of workspace interactions itself but in who records them: neither the reasoning of the agent (Weakness 1.1) nor a memory that the agent writes (Weakness 1.2) records them reliably, whereas RunningTab has the environment record every file that the agent reads or sees listed, and to the best of our knowledge, it is the first method that lets the environment keep this record.

Table C.1. Model-Kept Notebook on Workspace-Bench with the three LLMs (Pass Rate / TCR@70, three-run mean).

MethodGPT-5.4 nanoDeepSeek V4 FlashGemini 3.8 Flash
DCI41.63 / 20.6753.28 / 38.6731.42 / 23.33
Model-Kept Notebook41.78 / 21.6753.46 / 36.0041.33 / 29.00
RunningTab44.73 / 25.6759.99 / 41.0046.91 / 36.33

Table C.2. Read Log Only on Workspace-Bench with GPT-5.4 nano (three-run mean).

MethodRead log and candidatesRequirementsPass RateTCR@70
Read Log Onlycaptured by the environmentnot kept41.2521.67
RunningTabcaptured by the environmentkept44.7325.67

AuthorsOpenReview ↗

Response to Reviewer ZxnH (2/2)

Q1. Was thinking on, and if so, at which level?

Yes, thinking was on in all runs: GPT-5.4 nano runs with high reasoning effort, and DeepSeek V4 Flash and Gemini 3.8 Flash with their default thinking.


Q2. Does the content of the tab, or a subset of it, appear in the thinking tokens?

Mostly not. Using the audit described in our response to Weakness 1.1, we find that of the values that the tab would hold but the agent did not include in its deliverable, only 30.8% appear in the full reasoning of DeepSeek V4 Flash.


Q3. Does Pi keep a memory, and would the results differ with an agent that keeps one?

Pi keeps no memory beyond the context window, and its context management is designed to truncate long observations and clear older ones as they accumulate (Section 4). As for an agent that keeps a memory, Model-Kept Notebook gives the agent a memory outside the context window that it writes itself, and RunningTab outperforms it with all three LLMs (Table C.1), as detailed in our response to Weakness 1.2.


Q4. Why were the experiments not performed on larger models with more reasoning skills?

Following your suggestion, we have additionally run experiments with two larger models of the GPT-5.4 family, GPT-5.4 mini and the flagship GPT-5.4, both with high reasoning effort as with GPT-5.4 nano. As shown in Table C.3, RunningTab outperforms DCI with all three models on both metrics, including the flagship GPT-5.4. This indicates that RunningTab remains beneficial regardless of model size: even with stronger reasoning, the agent still benefits from having the environment keep track of every file that it reads or sees listed.

Table C.3. DCI and RunningTab on Workspace-Bench across the GPT-5.4 family (Pass Rate / TCR@70, three-run mean).

MethodGPT-5.4 nanoGPT-5.4 miniGPT-5.4
DCI41.63 / 20.6744.03 / 25.3349.53 / 30.33
RunningTab44.73 / 25.6746.28 / 27.6751.09 / 33.00

Dear Reviewer ZxnH,

As the discussion period closes on November 18 (AoE), we would like to gently remind you of our responses, which show that thinking keeps few of the values that the agent misses, that RunningTab outperforms two memory baselines, and that its gains hold with the larger GPT-5.4 mini and GPT-5.4. We would be deeply grateful if you could take a moment to review them. Your questions led to several of our new analyses, including the audit of thinking and the experiments with larger LLMs, and we sincerely hope that they address your concerns.

Warm regards, Authors

Reviewer_vKWXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RunningTab, an environment-maintained record that tracks task requirements, previously read content, and unopened files. It helps LLM agents avoid omissions in workspace deliverables. Experiments across three benchmarks and three LLMs show consistent improvements over the evaluated baselines.

Critical strengths
  1. The paper identifies a practically important failure mode: agents may read the correct information but fail to include it in the final deliverable after many subsequent interactions.
  2. The proposed design is simple and intuitive. Separating task requirements, read excerpts, and unopened candidates gives the agent a persistent representation of task progress.
Critical weaknesses
  1. Limited novelty. RunningTab combines persistent logging, requirement tracking, BM25 retrieval, and a final reminder. The paper does not clearly establish a substantial advance over existing memory and task-management approaches.
Questions for authors

How does RunningTab compare with a strong baseline that maintains a persistent requirement-and-evidence notebook outside the context window?

1 comment
AuthorsOpenReview ↗

Response to Reviewer vKWX

We sincerely thank you for your thoughtful and constructive review. We are also grateful that you found the failure mode practically important and the design simple and intuitive (giving the agent a persistent representation of task progress). We have carefully considered all of your comments and address each of them below with additional experiments.


W1. Limited novelty: RunningTab combines persistent logging, requirement tracking, BM25 retrieval, and a final reminder, without a clear advance over existing memory and task-management approaches.

Thank you for raising this point, and we would like to clarify that the novelty of RunningTab lies not in any of these components but in who records the task state: unlike existing memory and task-management approaches, which rely on the agent to write down what it reads, RunningTab has the environment record every file that the agent reads or sees listed. This matters because, in long-horizon tasks over many files, agents struggle to keep track of what they have read on their own, in two ways. First, even when the values they saw are still in their context, they leave many of them out of their deliverables (Section 3.1). Second, when asked to write these files down outside the context window, as in existing approaches, they record far fewer of them than the environment does: per attempt, the environment-side tab holds 3.7 to 26.5 times as many file entries as a notebook that the agent keeps itself (Model-Kept Notebook, Table B.2). These two findings indicate that the record of what the agent reads needs to be kept by the environment, which is exactly what RunningTab does, and to the best of our knowledge, it is the first method that lets the environment (not the agent) keep this record. As a result, RunningTab outperforms this notebook with all three LLMs (Table B.2).

Moreover, RunningTab is not a mere combination of the components that you listed: each of its two parts (the read log with BM25 retrieval and the requirements with a final reminder) is ineffective on its own, because what the agent lacks is neither a log of what it read nor a list of what the task requires, but a link between the two, and providing this link is a key and novel contribution of RunningTab. Specifically, with the read log but no requirements (Read Log Only), every file that the agent read is on record, but nothing indicates which of these files the task still needs, and as a result, the agent rarely goes back to the log (once in all 300 attempts with GPT-5.4 nano, against 4.0 times per task with RunningTab). Meanwhile, with the requirements but no read log (No Environmental Capture), the agent knows what is still open, but nothing points it to where the missing values are. Both variants therefore stay well below RunningTab (41.25 and 41.36 against 44.73; Table B.1). In contrast, RunningTab connects the two through the review and the evidence check: the review shows each open requirement beside the read-log entries that best match it, and the evidence check closes a requirement only when the agent cites a read-log entry as its evidence, which produces the gain.

Table B.1. Ablations of RunningTab on Workspace-Bench with GPT-5.4 nano (three-run mean).

MethodRead log and candidatesRequirementsPass RateTCR@70
Read Log Onlycaptured by the environmentnot kept41.2521.67
No Environmental Capturenot keptkept41.3622.33
RunningTabcaptured by the environmentkept44.7325.67

Q1. How does RunningTab compare with a strong baseline that keeps a persistent requirement-and-evidence notebook outside the context window?

Following your suggestion, we have evaluated Model-Kept Notebook, a persistent requirement-and-evidence notebook kept outside the context window that is identical to RunningTab (including every operation and check) except that the agent itself records the files that it reads or sees listed. Then, as shown in Table B.2, RunningTab outperforms this baseline with all three LLMs.

Table B.2. Model-Kept Notebook on Workspace-Bench with the three LLMs (Pass Rate / TCR@70, three-run mean).

MethodGPT-5.4 nanoDeepSeek V4 FlashGemini 3.8 Flash
DCI41.63 / 20.6753.28 / 38.6731.42 / 23.33
Model-Kept Notebook41.78 / 21.6753.46 / 36.0041.33 / 29.00
RunningTab44.73 / 25.6759.99 / 41.0046.91 / 36.33

Reviewer_wz5VOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper targets direct workspace interaction, where content the agent has already read often never reaches the deliverable. It proposes RunningTab, a task record kept by the environment outside the context window that stores requirements, read excerpts with provenance, and listed but unopened files, with checks at review, resolution and finish time. RunningTab outperforms DCI and three prompting baselines on three benchmarks with three LLMs.

Critical strengths

The diagnosis is concrete and quantified. On Workspace-Bench, about 19.8% of target values seen in observations are missing from deliverables, and the tab holds 94.8% of them.

The mechanism is simple, training free and model agnostic. The read log and candidates are captured deterministically by the harness without extra LLM calls, and the gains are consistent across three models.

The writing is clear. The formal definition is complete, and the figures convey the core idea and workflow intuitively.

Critical weaknesses

Gains may be confounded with level 3 context management, which truncates and clears older observations that the read log then stores and returns. Neither the clearing rate nor a control separating observation retention from task tracking is reported.

Baselines are weak and not information matched. All three are prompting methods, with no method that externalizes evidence bound task state and no context management method. Only the finish check of RunningTab lists unresolved requirements and unopened files.

No cost analysis. Four new operations and a status line appended every turn add tool calls, tokens and time, none of which are reported.

Limited novelty. Externalizing task state and checking a list before finishing are common ideas, and requirement decomposition and completion still rely on the model, so the contribution is closer to a harness feature.

Questions for authors
  1. How often is clearing triggered under level 3 for each method, and does the gain over DCI hold without clearing?
  2. How much gain does a variant achieve that keeps only the read log and retrieval, without requirements, review, evidence check and finish check?
  3. What essentially distinguishes RunningTab from prior methods that externalize evidence bound task state, and can a comparison be added?
  4. How many extra tool calls, tokens and how much extra time does RunningTab need over DCI?
4 comments
AuthorsOpenReview ↗

Response to Reviewer wz5V (1/3)

We sincerely thank you for your thorough review. We are also grateful that you found our diagnosis concrete and quantified; the mechanism simple, training-free, and model-agnostic; the gains consistent across three models; and the writing and figures clear. We have carefully considered all of your comments and address each of them below with additional analyses and new experiments.


W1. Gains may be confounded with level 3 context management (truncation and clearing), and neither the clearing rate nor a control (separating observation retention from task tracking) is reported.

Thank you for raising this concern, but we would like to clarify that the gain of RunningTab does not come from the truncation and clearing of observations, which we confirm below with both the clearing rate and the control that you described. First, we observe that clearing rarely occurs: on Workspace-Bench, it affects at most 4.7% of the attempts of any method and LLM, and RunningTab improves Pass Rate over DCI with all three LLMs both on the tasks where a DCI or RunningTab attempt was cleared and on the tasks where none was (for example, by 4.76 and 3.02 points with GPT-5.4 nano). Second, truncation does not explain the gain either. For both DCI and RunningTab, Pi keeps only the first 20,000 characters of a long observation, and the read log stores the observation in this truncated form, which means that the tab holds no content beyond these first 20,000 characters, the same content that DCI receives. Lastly, regarding the control that separates observation retention from task tracking, we have also run Read Log Only, which retains the observations in the read log that the environment captures but tracks no requirements. As shown in Table A.1, Read Log Only stays well below RunningTab, indicating that retaining observations alone does not produce the gain.

Table A.1. Read Log Only on Workspace-Bench with GPT-5.4 nano (three-run mean).

MethodRead log and candidatesRequirementsPass RateTCR@70
Read Log Onlycaptured by the environmentnot kept41.2521.67
RunningTabcaptured by the environmentkept44.7325.67

W2. The baselines are prompting only and not information matched, with no method that externalizes evidence-bound task state and no context management method.

Thank you for this suggestion. Following it, we have evaluated Model-Kept Notebook, an information-matched baseline, and RunningTab outperforms it with all three LLMs (Table A.2). Specifically, this baseline is identical to RunningTab, keeping the same task state outside the context window (the requirements, the files that the agent reads, and the files that it sees listed but does not open) with every operation and check, including the finish check. The only difference is who records the files: in RunningTab, the environment records every file that the agent reads or sees listed, whereas in this baseline, the agent has to record each of them in the notebook itself. This difference explains the advantage of RunningTab: the notebook that the agent keeps ends up far smaller than the tab, and more specifically, per attempt, the tab holds 3.7 to 26.5 times as many file entries as the notebook across the three LLMs.

As for a context management method, every method in our experiments already includes one: following DCI, all methods run with the context management of Pi at level 3 (Section 4), which truncates long observations and clears older ones as they accumulate. DCI is thus the agent with this context management but without the environment-side tab, and RunningTab improves over it with all three LLMs on all three benchmarks (Table 1).

Table A.2. Model-Kept Notebook on Workspace-Bench with the three LLMs (Pass Rate / TCR@70, three-run mean).

MethodGPT-5.4 nanoDeepSeek V4 FlashGemini 3.8 Flash
DCI41.63 / 20.6753.28 / 38.6731.42 / 23.33
Model-Kept Notebook41.78 / 21.6753.46 / 36.0041.33 / 29.00
RunningTab44.73 / 25.6759.99 / 41.0046.91 / 36.33

AuthorsOpenReview ↗

Response to Reviewer wz5V (2/3)

W3. The extra tool calls, tokens, and time that RunningTab needs over DCI are not reported.

Following your suggestion, we now report the cost of a Workspace-Bench task with DCI and RunningTab in Table A.3, and find that RunningTab adds only a small cost. Its extra agent cost is 0.03 to 0.12 USD per task with the three LLMs, since 80% to 90% of the extra input that RunningTab sends to the LLM repeats earlier context, which the LLM provider serves from its prompt cache at a fraction of the regular price. It also adds only 3 to 7 read and bash calls per task, the environment executes all tab operations of a task in under 7 seconds on average, and the median time per task rises by at most 0.7 minutes (with GPT-5.4 nano, a task in fact takes 1.1 minutes less than with DCI).

Table A.3. Cost per Workspace-Bench task (mean over three runs; median for minutes; input tokens include cached tokens).

LLMMethodRead and bash callsTab operationsInput tokensOutput tokensMedian minutesAgent cost (USD)Pass Rate
GPT-5.4 nanoDCI28.00544K14.7K4.80.03641.63
RunningTab31.615.11,215K20.3K3.70.06844.73
DeepSeek V4 FlashDCI31.80955K27.1K5.80.07153.28
RunningTab34.613.01,391K33.5K6.20.12359.99
Gemini 3.8 FlashDCI26.60608K7.9K1.70.20531.42
RunningTab33.65.61,349K11.2K2.40.32746.91

W4. Externalizing task state and checking a list before finishing are common ideas, and requirement decomposition and completion rely on the model, so the contribution is closer to a harness feature.

Thank you for this comment, and we would like to clarify that the novelty of RunningTab lies not in externalizing task state but in who records it: RunningTab has the environment record every file that the agent reads or sees listed, instead of relying on the agent to write these files down. This matters because, in long-horizon tasks over many files, agents struggle to keep track of what they have read on their own, in two ways. First, even when the values they saw are still in their context, they leave many of them out of their deliverables (Section 3.1). Second, when asked to record the files that they read and see listed outside the context window, they record far fewer of them than the environment does: per attempt, the environment-side tab holds 3.7 to 26.5 times as many file entries as a notebook that the agent keeps itself (Model-Kept Notebook, Table A.2). These two findings indicate that the record of what the agent reads needs to be kept by the environment, which is exactly what RunningTab does: it records these files in the tab and brings them back to each open requirement, and to the best of our knowledge, it is the first method that lets the environment (not the agent) keep this record.

To further experimentally examine your point that externalizing task state and checking a list before finishing are common ideas, we have compared RunningTab with two variants that keep these ideas but not the recording by the environment: No Environmental Capture, which keeps the requirements outside the context window with a finish check but records no files (Table 4 in our paper), and Model-Kept Notebook, which keeps the same task state and checks as RunningTab but has the agent record the files itself (Table A.2). As shown in these two tables, both variants stay well below RunningTab, indicating that these ideas alone do not produce the gain and that the gain comes from the recording by the environment. As for your other point on requirement decomposition and completion, the agent does state the requirements, but it cannot mark one as done without evidence from the environment: the evidence check accepts a requirement as done only when the agent cites an entry that the environment recorded in the read log. Removing this check lowers Pass Rate from 44.73 to 42.89 (No Evidence Check in Table 4), showing that completion is more reliable when the environment checks each requirement against the read log than when the model alone decides that a requirement is done.


AuthorsOpenReview ↗

Response to Reviewer wz5V (3/3)

Q1. How often is clearing triggered under level 3 for each method, and does the gain over DCI hold without clearing?

As detailed in our response to Weakness 1, clearing occurs in at most 4.7% of the attempts of each method and LLM, and RunningTab improves Pass Rate over DCI with all three LLMs both on the tasks with clearing and on the tasks without it.


Q2. How much does a variant that keeps only the read log and retrieval gain, without requirements, review, evidence check, and finish check?

As detailed in our response to Weakness 1, we have run this variant as Read Log Only, which keeps the environmental capture, the status line, and the two read operations of the tab (show and list) without the requirements, review, evidence check, or finish check. With GPT-5.4 nano on Workspace-Bench, it stays 3.48 points below RunningTab in Pass Rate (Table A.1), indicating that the read log and its retrieval alone are not enough to produce the gain.


Q3. What distinguishes RunningTab from prior methods that externalize evidence-bound task state, and can a comparison be added?

As detailed in our responses to Weaknesses 2 and 4, what distinguishes RunningTab is that the environment, not the agent, records every file that the agent reads or sees listed. For the comparison, we have evaluated Model-Kept Notebook, which keeps everything of RunningTab except that the agent records these files in its notebook, and RunningTab outperforms it with all three LLMs (Table A.2).


Q4. How many extra tool calls and tokens, and how much extra time, does RunningTab need over DCI?

As detailed in our response to Weakness 3, RunningTab adds only 0.03 to 0.12 USD of agent cost and 3 to 7 read and bash calls per task, and its median time per task rises by at most 0.7 minutes (Table A.3).


Dear Reviewer wz5V,

As the discussion period closes on November 18 (AoE), we would like to gently remind you of our responses, which report the clearing rate and the control that you asked for (Read Log Only), an information-matched baseline (Model-Kept Notebook), and the cost of RunningTab. We would be deeply grateful if you could take a moment to review them. Your detailed review led to several of our new analyses, including the Read Log Only control and the information-matched baseline, and we sincerely hope that they address your concerns.

Warm regards, Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (to all reviewers)

We would like to thank all reviewers for their time, effort, and constructive feedback. We are encouraged that the reviewers found the failure mode practically important (vKWX), our diagnosis concrete and quantified (wz5V), the design simple (wz5V, vKWX), the gains consistent across three models (wz5V), the experiments cleanly designed (ZxnH), and the paper clear and easy to follow (wz5V, ZxnH). As a reminder, RunningTab keeps a tab for each task outside the context window, which holds the requirements that the agent states and every file that the agent reads or sees listed. We summarize below how we have addressed the main concerns with new experiments and analyses.

  • Novelty: The novelty lies in who records these files: in RunningTab, the environment records them with no action from the agent, whereas existing memory and task-management approaches rely on the agent to write them down. The tab holds 3.7 to 26.5 times as many file entries as Model-Kept Notebook, which is identical to RunningTab except that the agent records the files itself, and RunningTab outperforms it.
  • Not a mere combination: Each of the two parts of RunningTab is ineffective on its own, because what the agent lacks is neither a log of what it read nor a list of what the task requires, but a link between the two, and providing this link is a key contribution of RunningTab. Keeping only the recorded files (Read Log Only) or only the requirements (No Environmental Capture) therefore stays well below RunningTab (Pass Rate of 41.25 and 41.36 against 44.73 with GPT-5.4 nano). In contrast, RunningTab connects the two through the review, which shows each open requirement beside the recorded files that best match it.
  • Context management: Clearing affects at most 4.7% of the attempts of any method and LLM, and RunningTab improves over DCI (the same agent without the tab) with all three LLMs on the tasks both with and without clearing. In addition, truncation does not explain the gain either, as the tab holds the same truncated observations that DCI receives.
  • Thinking and larger models: Thinking was on in all runs, yet the full reasoning of DeepSeek V4 Flash contains only 28.2% of the values that the agent saw but did not include in its deliverable. With the larger GPT-5.4 mini and flagship GPT-5.4, RunningTab still outperforms DCI on both metrics.
  • Efficiency: RunningTab adds only 0.03 to 0.12 USD of agent cost and 3 to 7 read and bash calls per Workspace-Bench task, and at most 0.7 minutes to the median time per task.

We have also updated our paper accordingly, with all changes highlighted in blue: Model-Kept Notebook (named Model-Kept Tab in the paper) in Table 1, Read Log Only in Table 4, the larger LLMs in Figure 6, and the analyses of context management, thinking, and efficiency in Appendix A. We fully understand that your time is valuable, and we would be deeply grateful if you could kindly take a moment to review our responses and the revised paper.

AuthorsOpenReview ↗

Gentle Reminder: The Discussion Period Closes on November 18 (AoE)

Dear Reviewers,

Thank you once again for your time and constructive feedback. We have carefully addressed every comment with new experiments and analyses, and have updated the paper accordingly (with all changes highlighted in blue). We fully understand that your time is extremely valuable, but as the discussion period closes on November 18 (AoE), we would be deeply grateful if you could kindly take a moment to review our responses. If they have addressed your concerns, we would sincerely appreciate it if you could consider reflecting this in your evaluation. Your feedback truly means a lot to us.

Warm regards, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.