- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes DWM, a framework that maintains workspace information and adds checks before and after an agent’s tool calls. It combines asset and relation tracking, action-outcome prediction, and artifact validation. Experiments across seven benchmarks and two backbones show improvements in task performance, with lower token costs in some settings.
Critical strengths
The paper addresses a practical problem: agents often spend substantial effort exploring workspaces and still fail to produce valid outputs. Integrating workspace state with execution checks is a useful contribution with some novelty. The improvements across both backbones are encouraging, particularly on DataSpace, DDR-Bench, and FDABench. The reduction in token usage also adds practical value.
Critical weaknesses
- “Data World Model” seems broader than what the paper establishes. The state representation and transition prediction are not developed in much detail, and the concrete mechanisms are mostly statistical checks and validation rules. The scope of the claim should be clearer.
- The static semantic-layer baseline does not isolate DWM’s contribution. Could an agent equipped with a checking skill, workspace-query tools, and a submission validator achieve similar gains? This comparison would help establish whether the full framework is necessary.
Questions for authors
- Which operations can the prediction module handle beyond joins, and how accurately does it predict their outcomes?
- How does DWM compare with skill- or tool-based checks under the same information and inference budget? Does automatic interception offer an advantage over agent-initiated checking?
- Could you remove Operation from the full system while retaining state updates and post-execution validation? The cumulative ablation does not show its contribution in the full framework.
4 comments
Response to Reviewer 83EC (PART 2/3)
Q3: Which operations can the prediction module handle beyond joins, and how accurately does it predict their outcomes?
A3: Thank you for asking about operations beyond joins. Operation predicts output attributes, including structure, size, and granularity, across projection, filtering, aggregation, and file operations. This lets the agent anticipate how different operations affect its data and deliverables.
(1) Operations and predicted attributes
- Projection and renaming: predict the output schema from the input columns and operation definition.
- Filtering and grouped aggregation: predict the schema, row-count interval, and output granularity.
- File operations: predict created, modified, or deleted files when their paths can be determined. Other attributes remain unknown.
(2) Prediction performance
Following your suggestion, we conducted an operation-level evaluation on AgenticDataBench and Workspace-Bench, using the same calls as the state-transition comparison in A1. Predictions are recorded before execution and checked against the actual results. Unknown attributes count as uncovered, and conditional accuracy is computed over calls with a prediction. For row counts, a prediction is correct when the observed count falls within the predicted interval.
| Operation and predicted attribute | Coverage (%) | Conditional accuracy (%) |
|---|---|---|
| Projection and renaming: output schema | 94.25 | 97.56 |
| Filtering: output schema | 87.84 | 96.92 |
| Filtering: output row-count interval | 68.92 | 92.16 |
| Grouped aggregation: output schema and granularity | 87.76 | 95.35 |
| Grouped aggregation: output row-count interval | 69.39 | 91.18 |
| File operations: expected change set | 89.47 | 94.12 |
The results show that Operation supports projection, filtering, aggregation, and file operations. Conditional accuracy remains high across these operations. For calls with predicted row-count intervals, more than 91% contain the observed count. Reporting coverage alongside accuracy also shows how often these predictions are available to the agent.
Thank you again for your suggestion. We will include the operation list, attribute-level results, and scoring rules.
Q4: Does automatic interception outperform agent-initiated skill or tool checking under matched information and inference budgets?
A4: Thank you for asking whether automatic interception adds value under matched conditions. We integrate checks with tool interactions so that state changes, risky actions, and submission receive timely feedback. The comparison below separates this triggering mechanism from access to checking capabilities.
(1) Task performance with automatic interception
Following your suggestion, we compared automatic interception with agent-initiated checking on AgenticDataBench and Workspace-Bench, using the same runs as A2. Both configurations have the same information, queries, updates, predictors, and validators. Only the triggering mechanism changes. The skill receives complete instructions and examples, and all internal calls count toward the same total budget.
| Triggering mechanism | AgenticDataBench score | Workspace-Bench score |
|---|---|---|
| Fully instructed checking skill | 49.08 | 52.00 |
| Automatic interception | 51.57 | 55.10 |
Automatic interception has higher task scores on both benchmarks in this comparison. Since capabilities and budgets are matched, these gains support the benefit of integrating checks into the tool loop.
(2) Overall interaction cost
We also measured end-to-end costs for the same AgenticDataBench runs. Tokens cover all agent and DWM model inputs and outputs. Elapsed time includes initialization, file measurements, execution, and feedback.
| Triggering mechanism | Mean tokens per task | Mean turns per task | Mean elapsed time (s/task) |
|---|---|---|---|
| Fully instructed checking skill | 44,782 | 29.7 | 182.6 |
| Automatic interception | 38,647 | 25.8 | 168.4 |
On AgenticDataBench, automatic interception improves task scores while reducing tokens, turns, and elapsed time under the same-capability control. For Workspace-Bench, the paper's interaction-cost figure reports a reduction from 186.0K to 43.0K tokens and from 8.76 to 5.91 turns relative to the original Baseline. These results support improved efficiency after accounting for DWM's model calls.
Thank you again for your suggestion. We will include the matched-budget comparison and the end-to-end cost accounting.
Response to Reviewer 83EC (PART 3/3)
Q5: What is the contribution of Operation when state updates and post-execution validation are retained?
A5: Thank you for proposing this independent ablation. Operation provides feedback about an action before it executes. Holding state updates and post-execution verification fixed allows us to isolate the value of that early feedback.
Following your suggestion, we conducted a controlled ablation of Operation on AgenticDataBench and Workspace-Bench. It disables only pre-execution prediction and its associated warnings and risk-based blocking. State access, continual updates, post-execution verification, and submission checks remain enabled. Both groups use the same model, tasks, initial state, and total budget.
| Configuration | AgenticDataBench score | Workspace-Bench score |
|---|---|---|
| Without Operation, retaining updates and verification | 49.03 | 51.80 |
| Full DWM | 51.57 | 55.10 |
In this supplementary ablation, full DWM scores higher on both benchmarks. Because state access and verification are unchanged, these gains support an additional benefit from pre-execution prediction. Operation helps the agent avoid problematic actions, while post-execution verification checks the artifacts actually produced.
Thank you again for your suggestion. We will include this controlled ablation and the component settings.
Thank you again for your careful review and valuable suggestions for improving our work. If our responses address your questions and resolve your concerns, we would appreciate it if you could reassess our work and consider increasing your rating.
Revised paper based on your suggestions
Thank you again for your careful and constructive review. We have uploaded a revised PDF, with the main additions highlighted in blue. In response to your comments:
- W1/Q1/Q3(State transitions):Clarified the formulation and evaluated predictions beyond joins.
- W2/Q2/Q4(Skill/tool comparisons):Added matched-capability comparisons and end-to-end cost measurements on AgenticDataBench.**
- Q5(Independent ablation):Isolated Operation’s contribution while retaining state updates and post-execution verification.**
We hope these response resolve your concerns. If so, we would appreciate it if you could consider increasing your rating. Please raise any remaining questions in the discussion, and we will respond promptly.
Response to Reviewer 83EC (PART 1/3)
Thank you for your careful review and valuable comments. We have refined the paper following your suggestions and respond to your questions below.
Q1: What supports the Data World Model formulation beyond statistical checks and validation rules, particularly its state representation and transition prediction?
A1: Thank you for asking us to clarify the Data World Model formulation. DWM models how an action changes the current data workspace and uses the predicted changes to guide the agent. Its state representation, action-conditioned predictions, and updates from observed execution form the basis of this formulation.
State records data assets, cross-file relationships, and generated artifacts. Given this state and an action, Operation predicts the output schema, row-count interval, granularity, and file changes. Task conditions determine whether a warning or rejection is warranted. After execution, Update incorporates the observed changes. The LLM interprets task semantics and forms checking conditions, while programmatic components measure data, compute supported transitions, and verify outcomes. For example, when an action creates an intermediate file, Update registers its measured schema. Subsequent operations can then be predicted from the updated workspace rather than the initial snapshot.
To address your concern about state representation and transition prediction, we conducted a comparison on AgenticDataBench and Workspace-Bench. The action and prediction rules are held fixed, while the input is either the initial state or the updated current state. Complete prediction success requires all pre-specified output attributes to be predicted correctly. Every call remains in the denominator, including calls with unknown attributes. This isolates whether maintaining the current state improves action-outcome prediction.
The updated state yields higher complete prediction success on both benchmarks. These gains support the role of continual observation in predicting workspace changes. A3 examines the predicted attributes by operation, and A5 isolates the effect of pre-execution prediction on task completion.
Thank you again for your suggestion. We will include the state-transition evaluation and clarify how prediction and updating use the evolving workspace state.
Q2: What does full DWM add beyond an agent equipped with a checking skill, workspace-query tools, and a submission validator?
A2: Thank you for suggesting this comparison. We agree that an agent should benefit from capable checking and query tools. DWM is designed to integrate these capabilities with the evolving workspace, providing feedback when the state changes, before risky actions, and at submission.
DWM connects these capabilities through three mechanisms:
Following your suggestion, we added a comparison on AgenticDataBench and Workspace-Bench in which the checking skill receives the same capabilities as DWM. Both share the initial state, task conditions, state-update functions, predictors, and validators. The skill receives complete instructions and examples and chooses when to call them, while DWM triggers them at the corresponding tool boundaries. Models and total budgets are matched, including all internal calls. This directly tests the benefit of automatic integration when the agent already has access to the full set of functions.
In this comparison, the fully instructed skill improves on basic tools, and full DWM provides further gains with lower token use. This supports the benefit of delivering the same capabilities at the relevant decision points. A4 compares the triggering mechanisms on the same task runs and reports end-to-end costs for AgenticDataBench.
Thank you again for your suggestion. We will include the same-capability comparison and its triggering rules in the revised version.