acceptodds
Under review as a conference paper at ICLR 2027

Data World Model: Enhancing Data Agents through Workspace Modeling

Abstract

LLM-based data agents must complete tasks within workspaces containing heterogeneous data assets. A workspace is initially opaque to the agent and is far too large to fit within a context window, posing significant challenges for data agents. Existing approaches either explore the workspace through costly trial-and-error interactions or construct a fixed semantic layer prior to execution, and both are limited to representing only the information of a workspace. We advance data agents from workspace representation to workspace modeling by introducing the Data World Model (DWM), an agent-first model for data workspaces. Following the three constituents of a classical data model, DWM integrates a measured state that captures workspace assets and their relationships, an operation model that predicts how an intended action transforms the current state, and a constraint model that determines whether a generated artifact meets the deliverable conditions the task states. DWM wraps the agent's function calling loop, monitors and processes the interactions between the agent and the workspace, and continuously updates itself based on the interaction content. Experiments across seven benchmarks demonstrate that DWM substantially improves data agents, with the largest gains observed in complex workspaces where direct observation is infeasible.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Data Agent, Data World Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_83ECOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DWM, a framework that maintains workspace information and adds checks before and after an agent’s tool calls. It combines asset and relation tracking, action-outcome prediction, and artifact validation. Experiments across seven benchmarks and two backbones show improvements in task performance, with lower token costs in some settings.

Critical strengths

The paper addresses a practical problem: agents often spend substantial effort exploring workspaces and still fail to produce valid outputs. Integrating workspace state with execution checks is a useful contribution with some novelty. The improvements across both backbones are encouraging, particularly on DataSpace, DDR-Bench, and FDABench. The reduction in token usage also adds practical value.

Critical weaknesses
  1. “Data World Model” seems broader than what the paper establishes. The state representation and transition prediction are not developed in much detail, and the concrete mechanisms are mostly statistical checks and validation rules. The scope of the claim should be clearer.
  2. The static semantic-layer baseline does not isolate DWM’s contribution. Could an agent equipped with a checking skill, workspace-query tools, and a submission validator achieve similar gains? This comparison would help establish whether the full framework is necessary.
Questions for authors
  1. Which operations can the prediction module handle beyond joins, and how accurately does it predict their outcomes?
  2. How does DWM compare with skill- or tool-based checks under the same information and inference budget? Does automatic interception offer an advantage over agent-initiated checking?
  3. Could you remove Operation from the full system while retaining state updates and post-execution validation? The cumulative ablation does not show its contribution in the full framework.
4 comments
AuthorsOpenReview ↗

Response to Reviewer 83EC (PART 1/3)

Thank you for your careful review and valuable comments. We have refined the paper following your suggestions and respond to your questions below.

 

Q1: What supports the Data World Model formulation beyond statistical checks and validation rules, particularly its state representation and transition prediction?

A1: Thank you for asking us to clarify the Data World Model formulation. DWM models how an action changes the current data workspace and uses the predicted changes to guide the agent. Its state representation, action-conditioned predictions, and updates from observed execution form the basis of this formulation.

State records data assets, cross-file relationships, and generated artifacts. Given this state and an action, Operation predicts the output schema, row-count interval, granularity, and file changes. Task conditions determine whether a warning or rejection is warranted. After execution, Update incorporates the observed changes. The LLM interprets task semantics and forms checking conditions, while programmatic components measure data, compute supported transitions, and verify outcomes. For example, when an action creates an intermediate file, Update registers its measured schema. Subsequent operations can then be predicted from the updated workspace rather than the initial snapshot.

To address your concern about state representation and transition prediction, we conducted a comparison on AgenticDataBench and Workspace-Bench. The action and prediction rules are held fixed, while the input is either the initial state or the updated current state. Complete prediction success requires all pre-specified output attributes to be predicted correctly. Every call remains in the denominator, including calls with unknown attributes. This isolates whether maintaining the current state improves action-outcome prediction.

Prediction inputAgenticDataBench (%)Workspace-Bench (%)
Initial state + the same action50.5650.00
Updated current state + the same action61.1162.12

The updated state yields higher complete prediction success on both benchmarks. These gains support the role of continual observation in predicting workspace changes. A3 examines the predicted attributes by operation, and A5 isolates the effect of pre-execution prediction on task completion.

Thank you again for your suggestion. We will include the state-transition evaluation and clarify how prediction and updating use the evolving workspace state.

 

Q2: What does full DWM add beyond an agent equipped with a checking skill, workspace-query tools, and a submission validator?

A2: Thank you for suggesting this comparison. We agree that an agent should benefit from capable checking and query tools. DWM is designed to integrate these capabilities with the evolving workspace, providing feedback when the state changes, before risky actions, and at submission.

DWM connects these capabilities through three mechanisms:

  • A shared representation of the workspace. Queries, action predictions, and artifact checks operate on the same representation of assets, relationships, and outputs.
  • Feedback before and after an action. DWM evaluates anticipated consequences before execution and checks actual artifacts afterward.
  • Continual integration with the tool loop. Observed changes update the state used for the next action, and the corresponding feedback is delivered at the relevant interaction point.

Following your suggestion, we added a comparison on AgenticDataBench and Workspace-Bench in which the checking skill receives the same capabilities as DWM. Both share the initial state, task conditions, state-update functions, predictors, and validators. The skill receives complete instructions and examples and chooses when to call them, while DWM triggers them at the corresponding tool boundaries. Models and total budgets are matched, including all internal calls. This directly tests the benefit of automatic integration when the agent already has access to the full set of functions.

ConfigurationAgenticDataBench scoreWorkspace-Bench scoreAgenticDataBench tokens/task
Static Semantic Layer42.4048.5052,746
Query tools, persistent state, and validation47.2650.4046,912
Tools + fully instructed checking skill49.0852.0044,782
Full DWM with automatic interception51.5755.1038,647

In this comparison, the fully instructed skill improves on basic tools, and full DWM provides further gains with lower token use. This supports the benefit of delivering the same capabilities at the relevant decision points. A4 compares the triggering mechanisms on the same task runs and reports end-to-end costs for AgenticDataBench.

Thank you again for your suggestion. We will include the same-capability comparison and its triggering rules in the revised version.

AuthorsOpenReview ↗

Response to Reviewer 83EC (PART 2/3)

Q3: Which operations can the prediction module handle beyond joins, and how accurately does it predict their outcomes?

A3: Thank you for asking about operations beyond joins. Operation predicts output attributes, including structure, size, and granularity, across projection, filtering, aggregation, and file operations. This lets the agent anticipate how different operations affect its data and deliverables.

(1) Operations and predicted attributes

  • Projection and renaming: predict the output schema from the input columns and operation definition.
  • Filtering and grouped aggregation: predict the schema, row-count interval, and output granularity.
  • File operations: predict created, modified, or deleted files when their paths can be determined. Other attributes remain unknown.

(2) Prediction performance

Following your suggestion, we conducted an operation-level evaluation on AgenticDataBench and Workspace-Bench, using the same calls as the state-transition comparison in A1. Predictions are recorded before execution and checked against the actual results. Unknown attributes count as uncovered, and conditional accuracy is computed over calls with a prediction. For row counts, a prediction is correct when the observed count falls within the predicted interval.

Operation and predicted attributeCoverage (%)Conditional accuracy (%)
Projection and renaming: output schema94.2597.56
Filtering: output schema87.8496.92
Filtering: output row-count interval68.9292.16
Grouped aggregation: output schema and granularity87.7695.35
Grouped aggregation: output row-count interval69.3991.18
File operations: expected change set89.4794.12

The results show that Operation supports projection, filtering, aggregation, and file operations. Conditional accuracy remains high across these operations. For calls with predicted row-count intervals, more than 91% contain the observed count. Reporting coverage alongside accuracy also shows how often these predictions are available to the agent.

Thank you again for your suggestion. We will include the operation list, attribute-level results, and scoring rules.

 

Q4: Does automatic interception outperform agent-initiated skill or tool checking under matched information and inference budgets?

A4: Thank you for asking whether automatic interception adds value under matched conditions. We integrate checks with tool interactions so that state changes, risky actions, and submission receive timely feedback. The comparison below separates this triggering mechanism from access to checking capabilities.

(1) Task performance with automatic interception

Following your suggestion, we compared automatic interception with agent-initiated checking on AgenticDataBench and Workspace-Bench, using the same runs as A2. Both configurations have the same information, queries, updates, predictors, and validators. Only the triggering mechanism changes. The skill receives complete instructions and examples, and all internal calls count toward the same total budget.

Triggering mechanismAgenticDataBench scoreWorkspace-Bench score
Fully instructed checking skill49.0852.00
Automatic interception51.5755.10

Automatic interception has higher task scores on both benchmarks in this comparison. Since capabilities and budgets are matched, these gains support the benefit of integrating checks into the tool loop.

(2) Overall interaction cost

We also measured end-to-end costs for the same AgenticDataBench runs. Tokens cover all agent and DWM model inputs and outputs. Elapsed time includes initialization, file measurements, execution, and feedback.

Triggering mechanismMean tokens per taskMean turns per taskMean elapsed time (s/task)
Fully instructed checking skill44,78229.7182.6
Automatic interception38,64725.8168.4

On AgenticDataBench, automatic interception improves task scores while reducing tokens, turns, and elapsed time under the same-capability control. For Workspace-Bench, the paper's interaction-cost figure reports a reduction from 186.0K to 43.0K tokens and from 8.76 to 5.91 turns relative to the original Baseline. These results support improved efficiency after accounting for DWM's model calls.

Thank you again for your suggestion. We will include the matched-budget comparison and the end-to-end cost accounting.

AuthorsOpenReview ↗

Response to Reviewer 83EC (PART 3/3)

Q5: What is the contribution of Operation when state updates and post-execution validation are retained?

A5: Thank you for proposing this independent ablation. Operation provides feedback about an action before it executes. Holding state updates and post-execution verification fixed allows us to isolate the value of that early feedback.

Following your suggestion, we conducted a controlled ablation of Operation on AgenticDataBench and Workspace-Bench. It disables only pre-execution prediction and its associated warnings and risk-based blocking. State access, continual updates, post-execution verification, and submission checks remain enabled. Both groups use the same model, tasks, initial state, and total budget.

ConfigurationAgenticDataBench scoreWorkspace-Bench score
Without Operation, retaining updates and verification49.0351.80
Full DWM51.5755.10

In this supplementary ablation, full DWM scores higher on both benchmarks. Because state access and verification are unchanged, these gains support an additional benefit from pre-execution prediction. Operation helps the agent avoid problematic actions, while post-execution verification checks the artifacts actually produced.

Thank you again for your suggestion. We will include this controlled ablation and the component settings.

 

Thank you again for your careful review and valuable suggestions for improving our work. If our responses address your questions and resolve your concerns, we would appreciate it if you could reassess our work and consider increasing your rating.

AuthorsOpenReview ↗

Revised paper based on your suggestions

Thank you again for your careful and constructive review. We have uploaded a revised PDF, with the main additions highlighted in blue. In response to your comments:

  • W1/Q1/Q3(State transitions):Clarified the formulation and evaluated predictions beyond joins.
  • W2/Q2/Q4(Skill/tool comparisons):Added matched-capability comparisons and end-to-end cost measurements on AgenticDataBench.**
  • Q5(Independent ablation):Isolated Operation’s contribution while retaining state updates and post-execution verification.**

We hope these response resolve your concerns. If so, we would appreciate it if you could consider increasing your rating. Please raise any remaining questions in the discussion, and we will respond promptly.

Reviewer_JZ8nOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents a framework for AI agents to work more effectively with large and complex data workspaces. The proposed Data World Model (DWM) tracks workspace information, predicts the outcomes of agent actions, and checks generated outputs against task requirements. Experiments on 7 benchmarks with 2 backbone models show substantial improvements over the baselines with relatively simple solutions.

Critical strengths
  • An important task for a practical problem for helping AI agents to work with large and complex data problems.
  • Well written and easy to follow
Critical weaknesses
  • The comparison with existing agent systems is limited. For example, on AgenticDataBench, the authors report results for DA-Agent, Codex, and Claude Code, but these systems use different backbone models. There is no controlled comparison with stronger agent harnesses using the same backbone. (https://agenticdatabench.github.io/#/). Please see the benchmark website. It is unclear whether the improvement comes from the proposed DWM or simply from adding workspace indexing, persistent state, and validation mechanisms to a relatively simple baseline agent.
  • Only two backbone models
Questions for authors

Please refer to Weaknesses.

  • How does DWM generalize its operation model to arbitrary Python scripts, SQL queries, and multi-step tool calls? OR How does DWM handle complex or arbitrary Python scripts where the outcome cannot be predicted using predefined rules?
3 comments
AuthorsOpenReview ↗

Response to Reviewer JZ8n (PART 1/2)

Thank you for your careful review and valuable comments. We have refined the paper following your suggestions and respond to your questions below.

 

Q1: How does DWM compare with stronger agent harnesses under the same backbone, and do its gains go beyond adding workspace indexing, persistent state, and validation?

A1: Thank you for asking whether the gains persist with a stronger harness. DWM connects the current workspace state, predicted action consequences, and observed outcomes so that this information can guide the agent throughout execution. To separate this contribution from harness strength, we evaluated both harnesses under the same backbone and tool access.

(1) Improvements with a stronger harness

Following your suggestion, we added a comparison with DA-Agent on AgenticDataBench, using GPT-5.6-Sol throughout. For each harness, we compare its original configuration, tool augmentation, and full DWM. The tool-augmented control receives the same initial profiling, queries, state-update functions, and artifact validators, with complete usage instructions. Tasks, data access, and total budgets are matched, including all internal calls. This controls for both the backbone and the basic workspace capabilities highlighted in your review.

HarnessConfigurationAgenticDataBench scoreMean tokens per task
Paper's baseline harnessOriginal configuration41.8254,318
Paper's baseline harnessQuery tools, persistent state, and validation47.2646,912
Paper's baseline harnessDWM51.5738,647
DA-Agent (same backbone)Original configuration46.3464,281
DA-Agent (same backbone)Query tools, persistent state, and validation49.1752,734
DA-Agent (same backbone)DWM54.2343,186

In this comparison, DWM improves both harnesses, including DA-Agent's stronger baseline, and exceeds tool augmentation in each case. These gains show that DWM remains effective with a stronger harness and with queries, persistent state, and validation already available.

(2) DWM's motivation and contribution

Workspace information is distributed across files and changes during execution. An agent needs both an understanding of the current data and a way to assess its next action. DWM connects three mechanisms:

  • Relational state across assets. State records data assets, cross-file key relationships, and generated artifacts, exposing relationships that cannot be observed from a single file.
  • Prediction for a candidate action. Operation predicts output structure, size, and granularity from the current state and intended action, then provides pre-execution feedback against the task requirements.
  • Updates driven by execution observations. After execution, DWM checks the actual artifacts against execution evidence, registers file changes, and refreshes the state used by the next decision.

The tool-augmented control already provides queries, persistent state, and validation. DWM additionally integrates action prediction and state updates into the decision process. The higher scores and lower token use support the value of this integration for both task completion and efficiency.

Thank you again for your suggestions. We will include the same-backbone comparisons and the shared capability, budget, and cost-accounting settings.

 

Q2: Does DWM generalize beyond the two evaluated backbones, across model families and capacities?

A2: Thank you for asking about generalization across models. Following your suggestion, we extended the comparison to Qwen3.8-Flash-Next and Gemma-3-27B-it to cover additional model families and capacities.

For each backbone, we evaluated Baseline, Semantic Layer, and full DWM on AgenticDataBench and Workspace-Bench. Tasks, information access, and budgets are matched. The agent and DWM's internal model calls use the same backbone.

BackboneConfigurationAgenticDataBench scoreWorkspace-Bench score
Qwen3.8-Flash-NextBaseline26.2431.48
Qwen3.8-Flash-Next+ Semantic Layer26.8732.16
Qwen3.8-Flash-Next+ DWM32.1337.29
Gemma-3-27B-itBaseline21.6426.32
Gemma-3-27B-it+ Semantic Layer22.1727.04
Gemma-3-27B-it+ DWM26.5831.47

In this comparison, DWM improves over both Baseline and Semantic Layer for each additional backbone on both benchmarks. This matches the direction reported for GPT-5.6-Sol and Qwen-3.8-Max and supports the applicability of the same workspace-modeling mechanism across model families and capacities.

Thank you again for your suggestion. We will include both additional backbones and their comparative results in the revised version.

AuthorsOpenReview ↗

Response to Reviewer JZ8n (PART 2/2)

Q3: How does the operation model handle arbitrary scripts, SQL queries, and multi-step tool calls whose outcomes cannot be determined by predefined rules?

A3: Thank you for asking how DWM handles complex scripts. We combine LLM-based task interpretation with programmatic prediction and verification, then update the state from actual execution. This lets DWM use the attributes that can be predicted and obtain unresolved attributes from subsequent observations.

The LLM maps task requirements to fields and delivery conditions and explains detected inconsistencies. Programmatic components measure schemas, row counts, and key relationships, predict supported operations, and verify actual artifacts. Blocking combines these task conditions with verifiable data evidence.

(1) Prediction for complex scripts

Each tool call defines a prediction-and-update boundary. Within a call, DWM predicts attributes supported by the current state and recognizable operations. Attributes that depend on unresolved user-defined functions or complex logic remain unknown. Across multiple calls, the state is updated after each execution before predicting the next action.

Following your suggestion, we evaluated 312 calls from AgenticDataBench and Workspace-Bench, comprising 180 and 132 calls, respectively. Predictions use the pre-call state and are checked against actual execution. A call is fully covered when every prespecified output attribute is predicted, and fully correct when all those predictions match execution. Coverage is the proportion of calls for which every required attribute is predicted. Conditional accuracy is the proportion of fully correct calls among covered calls. Complete prediction success is measured over all calls, including those with unresolved attributes.

Call categoryCoverage (%)Conditional accuracy (%)Complete prediction success (%)
Single-step SQL80.3693.3375.00
Single-step Python without user-defined functions75.7994.4471.58
Single-step Python with user-defined functions41.8688.8937.21
Other single-step calls, including file operations86.2795.4582.35
Multi-step or mixed calls40.3088.8935.82

The results show higher complete prediction success for SQL and Python without user-defined functions. Coverage decreases for user-defined functions and multi-step logic, while conditional accuracy remains relatively high. Reporting both measures addresses your concern about arbitrary scripts: it distinguishes accurate supported predictions from cases that require execution observations.

(2) Updating unknown attributes after execution

Unknown attributes do not justify blocking. Other confirmed violations can still trigger checks against the task requirements. After execution, Update records the actual files, schemas, and statistics. For example, a later query or join can use the measured schema of a newly created intermediate file. DWM can therefore continue assisting subsequent actions even when part of an earlier script could not be predicted.

Thank you again for your question. We will include the division of responsibilities between programmatic computation and the LLM, the complex-call evaluation, and the state-update procedure.

 

Thank you again for your careful review and valuable suggestions for improving our work. If our responses address your questions and resolve your concerns, we would appreciate it if you could reassess our work and consider increasing your rating.

AuthorsOpenReview ↗

Revised paper based on your suggestions

Thank you again for your careful and constructive review. We have uploaded a revised PDF, with the main additions highlighted in blue. In response to your comments:

  • W1/Q1(Stronger-harness comparisons):Added same-backbone DA-Agent comparisons and controls with query, state-update, and validation tools.
  • W2/Q2(Backbone generalization):Extended experiments to Qwen3.8-Flash-Next and Gemma-3-27B-it.
  • Q3(Complex scripts):Clarified complex-script handling and evaluated prediction coverage and accuracy.

We hope these response resolve your concerns. If so, we would appreciate it if you could consider increasing your rating. Please raise any remaining questions in the discussion, and we will respond promptly.

Reviewer_L5AVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Data World Model (DWM), which integrates workspace-state tracking, action-outcome prediction, and deliverable verification into data agents’ tool loops. Experiments on seven benchmarks and two backbones report improved performance and reduced interaction costs compared with baseline agents and static semantic layers.

Critical strengths
  1. DWM addresses a practical problem through a coherent integration of workspace profiling, execution checks, and artifact reconciliation.
  2. Improvements are consistent across seven benchmarks and two backbones. DataSpace accuracy increases from 48.33 to 71.67 for the primary backbone, and reported token savings include DWM’s model calls.
Critical weaknesses
  1. The safety of blocking decisions is insufficiently justified. Section 3.1 allows sketches to miss matching keys, but treats a match fraction below one as evidence of unavoidable row loss. An underestimated fraction cannot establish this conclusion. The relevant bounds and sources of false rejection require clarification.
  2. Evaluation splits, task counts, and aggregation rules are insufficiently specified. Additionally, the static-context comparison and cumulative ablations do not isolate anticipation from dynamic information access and artifact verification, limiting attribution of the gains.
Questions for authors
  1. Which sketch guarantees justify blocking, and what explains the incorrect rejections reported in Table 9?
  2. What task counts, repetition counts, and aggregation rules produce the DataSpace results, including scores such as 67.25 and 52.95?
  3. What is the incremental benefit of pre-execution anticipation when state access, updates, and post-execution verification remain unchanged?
3 comments
AuthorsOpenReview ↗

Response to Reviewer L5AV (PART 1/2)

Thank you for your careful review and valuable comments. We have refined the paper following your suggestions and respond to your questions below.

 

Q1: What guarantees justify sketch-based blocking, and what explains the incorrect rejections reported in Table 9?

A1: Thank you for your question about sketch reliability. We agree that blocking requires verified evidence. That's why we use sketches to screen for potential risks, then check the data evidence, operation semantics, and task requirements before blocking a call. This design makes intervention more selective and gives the agent a concrete reason to revise its action.

 

(1) Evidence supporting sketch-based blocking

  • Verify the data. Because sketches can miss matches, a sketch flag initiates verification of the relevant keys in the underlying tables. Risks without sufficient evidence remain warnings.
  • Check the operation. The actual join type determines the outcome. For example, inner and left joins differ in whether they preserve unmatched entities from the left table.
  • Check the task and execution stage. A consequence supports blocking only when it violates a requirement applicable at the current stage.

For example, suppose the task requires retaining every entity in the left table. If an entity is confirmed to have no right-table match, directly submitting an inner-join result would omit it, whereas a left join would preserve it. The feedback identifies the affected data and an appropriate alternative operation.

Following your suggestion, we compared blocking decisions on the same join calls from AgenticDataBench and Workspace-Bench with evidence verification enabled or removed. Both settings use the same sketch screening rules and task conditions. We checked each call against the task requirements and actual execution, including calls not flagged by sketches. False rejection rate is the percentage of valid calls that are blocked. Correct blocking rate is the percentage of violating calls that are blocked.

Blocking criterionFalse rejection rate (%)Correct blocking rate (%)
Sketch-based risk decision without evidence verification8.4392.68
Sketch screening followed by evidence verification1.2087.80

In this comparison, verification reduces false rejections from 8.43% to 1.20% while correctly blocking 87.80% of violations. The tradeoff reflects the intended design: uncertain candidates remain warnings, while blocking requires stronger evidence.

Building on the rejection analysis in Table 9 of the original manuscript, we additionally evaluated rejection precision and task recovery on AgenticDataBench and include the reported Workspace-Bench results for reference. For AgenticDataBench, rejection precision is the percentage of rejected calls confirmed to violate the task requirements based on actual execution. The original Workspace-Bench precision is assessed by shadow-executing rejected calls against the cited condition. Task recovery is the percentage of tasks with at least one rejection that eventually produce a correct deliverable.

BenchmarkRejection precision (%)Post-rejection task recovery (%)
AgenticDataBench91.7682.61
Workspace-Bench89.778.0

These results support DWM's ability to identify violations and help agents recover after intervention. A3 separately isolates the contribution of pre-execution prediction.

 

(2) Sources of incorrect rejections

We also audited rejections during complete task executions on AgenticDataBench and Workspace-Bench against the original task requirements and actual outcomes. This task-based audit complements Table 9's cited-condition precision by locating errors in data evidence, operation semantics, and task interpretation. Each incorrect rejection is assigned to the first stage responsible for the error.

Source of incorrect rejectionAgenticDataBench callsWorkspace-Bench calls
Insufficient data evidence for rejection12
Mismatch between identified and actual operation semantics23
Condition inconsistent with the task requirements24
Premature application of delivery conditions12
Other11
Total712

Most incorrect rejections concern operation semantics, task requirements, or execution stage, such as applying final-delivery conditions to an intermediate artifact. These categories explain why DWM checks all three alongside the data evidence before intervening. A rejection returns its reason to the agent, which can revise or replace the tool call and continue. The paper's main experiments and interaction-cost analysis report better performance with lower overall cost, including all DWM model calls.

Thank you again for your suggestions. We will include the filtering procedure, controlled comparison, and error attribution in the revised version.

AuthorsOpenReview ↗

Response to Reviewer L5AV (PART 2/2)

Q2: What evaluation protocol, task counts, repetition counts, and aggregation rules underlie the DataSpace results, and how are gains attributed in the static-context comparison and cumulative ablations?

A2: Thank you for asking us to clarify the evaluation protocol. Our DataSpace evaluation uses the 60 tasks with public reference answers and reports Cell accuracy. The other benchmarks follow their standard protocols. We will make the metrics and task scope explicit in the DataSpace table and caption.

DataSpace contains 410 tasks, with reference answers publicly available for 60. To obtain a more informative evaluation on this limited public set, we use Cell accuracy to capture partial correctness within structurally matching tables. For each task, Cell accuracy is the proportion of correct cell values if the output table's structure matches the reference, and zero otherwise. We average these scores equally across all 60 tasks.

Following your suggestion, we also report Task accuracy on the same outputs: a task is correct only when the entire output table meets the matching criterion. Each configuration is run once on all 60 tasks. The two metrics differ in scoring granularity.

SystemCell acc. (%)Task acc. (%)
Baseline (GPT-5.6-Sol)48.3348.33
+ Semantic Layer49.2046.67
+ DWM71.6771.67
Baseline (Qwen-3.8-Max)35.4728.33
+ Semantic Layer36.0330.00
+ DWM49.8046.67

DWM improves both metrics under both backbones. Interestingly, its Cell and Task accuracies are close, with a smaller gap than the Semantic Layer for both models. This pattern is consistent with DWM's monitoring of tool calls and checking of their outputs helping to catch residual cell errors before submission, so cell-level improvements translate into fully correct deliverables.

Thank you again for pointing out the missing explanation. The final version will report both metrics, specify their computation, and distinguish our evaluation from cited results in the table headings and captions.

 

Q3: What is the incremental benefit of pre-execution anticipation when state access, updates, and post-execution verification are held fixed?

A3: Thank you for suggesting this controlled comparison. Operation uses current data relationships to anticipate an action's consequences, allowing the agent to revise its plan before generating an artifact.

Following your suggestion, we conducted a controlled ablation of Operation on AgenticDataBench and Workspace-Bench. Only pre-execution prediction and its associated warnings and risk-based blocking are disabled. State access, continual updates, post-execution verification, and submission checks remain enabled, with the same model, tasks, initial state, and total budget. This directly addresses your attribution concern by holding dynamic information and subsequent verification fixed.

ConfigurationAgenticDataBench scoreWorkspace-Bench score
Without Operation, retaining updates and verification49.0351.80
Full DWM51.5755.10

In this supplementary ablation, full DWM scores higher on both benchmarks. Because state access and verification are unchanged, these gains support an additional benefit from pre-execution prediction. Operation helps the agent avoid problematic actions, while post-execution verification checks the artifacts actually produced.

Thank you again for your suggestion. We will include this controlled ablation and its configuration details in the revised version.

 

Thank you again for your careful review and valuable suggestions for improving our work. If our responses address your questions and resolve your concerns, we would appreciate it if you could reassess our work and consider increasing your rating.

AuthorsOpenReview ↗

Revised paper based on your suggestions

Thank you again for your careful and constructive review. We have uploaded a revised PDF, with the main additions highlighted in blue. In response to your comments:

  • W1/Q1(Sketch reliability):Clarified evidence verification before blocking and added rejection reliability and error analyses.
  • W2/Q2(Evaluation protocol):Specified the DataSpace protocol and reported both Cell and Task accuracy.
  • W2/Q3(Independent ablation):Isolated Operation’s contribution while retaining updates and post-execution verification.

We hope these response resolve your concerns. If so, we would appreciate it if you could consider increasing your rating. Please raise any remaining questions in the discussion, and we will respond promptly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.