acceptodds
Withdrawn from ICLR 2027

StateLedger: Dependency-Closed Analytical State for Executable Data-Analysis Replay

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models are increasingly used for multi-step data-analysis workflows in which analytical state accumulates across cleaned tables, repaired schemas, intermediate variables, and derived metrics. When such history is stored or retrieved as undifferentiated text, however, a later computation may omit prerequisite transformations or recover the wrong version of a variable, preventing faithful replay. We present StateLedger, which selects executable history through dependency closure over versioned analytical state. StateLedger represents prior analysis as dependency-linked state events, resolves the state versions required by target code, recursively recovers their prerequisite producers, and projects the resulting closure back onto a chronological subsequence of whole historical turns. The result is a reduced history intended to preserve the state required for execution. On 455 stable CoCoNote notebook targets, its source-only instantiation achieves 342 exact replays while selecting 38.4% fewer source characters on average than the full prefix. It substantially outperforms character-budget-matched BM25 and plain static slicing. Its gain over qualified reaching slicing is modest, with notebook-level uncertainty including zero, while prefix-instrumented dynamic recovery is more accurate at similar selected-history cost. With the recent-three guard fixed, adding transitive prerequisite expansion raises exact replay from 25/455 to 339/455, isolating the principal mechanism beyond direct producer recovery. An external notebook-trace pilot and target-code-hidden generation diagnostic provide complementary evidence. Together, these findings support dependency-closed analytical state as a practical unit for repair-free replay from reduced analytical history.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, LLM agents, data analysis agents, structured memory, dependency tracking, executable replay, workflow reconstruction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_kumXOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

StateLedger selects historical code needed to rerun a given data-analysis computation. It tracks versions of variables and columns, follows their dependencies backwards, and replays the selected cells in their original order. The main evaluation measures whether this reduced history reproduces the output obtained using the full history.

Critical strengths
  • Testing actual execution gives a concrete way to evaluate whether the selected history is sufficient.
  • The authors include stronger slicing baselines and openly report negative results, including the lack of a clear gain in the fully online setting.
Critical weaknesses
  1. The practical motivation is not established. The introduction describes accumulating analytical state and lists related benchmarks, but never clearly explains the workflow in which selective replay is needed. What makes keeping the existing execution state or rerunning the full history impractical? The connection between this problem and LLM memory is also weak: the main experiment assumes that the target code is already known. I understand the evaluation task, but I am not convinced why it is an important research problem in the setting studied.

  2. The writing makes a fairly straightforward idea difficult to follow. Terms such as “versioned analytical state” “producer resolution” and “executable projection” appear before a concrete example establishes what is going wrong. The example in Section 4.4 should have appeared much earlier. The abstract is also fragmented into blocks and reads like a compressed experimental report, with too many numbers and qualifications before the reader understands the practical goal.

  3. The methodological contribution over program slicing is unclear. Tracking variable versions and recursively recovering dependencies are familiar ideas. The paper itself acknowledges its connection to cell-level reaching-definition analysis, but does not clearly identify what the ledger representation adds. The closure ablation shows that indirect dependencies matter; it does not establish that the proposed representation is needed. Likewise, the replay guarantee assumes complete and faithful provenance, leaving the reliability of the actual extraction procedure unresolved.

  4. The recent-three guard is poorly justified. It adds cells without necessarily recovering their dependencies. In Table 2, removing it improves replay accuracy from 74.5% to 80.0% while also reducing history size. This raises a basic question about the design of the default method.

  5. The evaluation leaves the claimed practical benefit uncertain. Only 455 of 1050 CoCoNote candidates are retained, and their histories average about 1781 source characters. The filtering is method-independent, but it favors shorter histories and limits the evidence for long-history use. Reducing source characters alone also does not establish a worthwhile end-to-end saving once selection, extraction, and replay failures are considered.

Questions for authors
  1. What specific deployment scenario requires selective replay, and why are retaining execution state or replaying the full history inadequate there?
  2. What capability does StateLedger add beyond a version-aware, column-aware backward slicer under the same information constraints?
  3. Why is recent-three the default when it hurts both accuracy and history size on the main benchmark? How do the stronger baselines compare without this guard?
  4. Can you demonstrate a practical benefit on longer histories, including total overhead and downstream task success, rather than primarily selected source length?
Reviewer_q438OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents StateLedger, a method for recovering the earlier code needed to rerun a particular analysis step. It represents prior computations as versioned state events and recursively traces the target's dependencies to select the historical code turns needed for replay. The method is evaluated on 455 CoCoNote targets, and 342 among them are exact-replayed, performing better than qualified reaching slicing, BM25, plain static slicing, and recent3 baselines.

Critical strengths
  1. The paper addresses a practical problem and clearly formulates executable history recovery through versioned state dependencies.
  2. The evaluation includes matched-budget comparisons and controlled ablations, with transparent reporting of uncertainty and negative results.
Critical weaknesses
  1. Limited technical novelty. The method is straightforward, combining version-aware dependency tracking with backward closure. The paper does not clearly establish its technical advance over existing program slicing methods, such as Head et al. (2019) and nbslicer (Shankar et al., 2022). The paper should clearly explain what capability and contribution the proposed mechanisms (particularly the event representation) add over these works.

  2. Limited empirical advantages over baselines. StateLedger vs. qualified slicing's notebook-level marco confidence interval includes zero (Section 6.1), and StateLedger performs worse than qualified slicing and BM25 on LongDS (Table 7). The improvements over the baselines are not consistent. An analysis of the cases where StateLedger uniquely succeeds may help explain its value.

Questions for authors
  1. How does StateLedger perform without state versioning?
  2. The selected CoCoNote cases comprise much shorter histories. How does StateLedger perform on longer and more difficult histories?
  3. Retained CoCoNote histories average only 1781 characters. In what realistic setting is the full prefix replay infeasible, so that a 25% replay-failure rate is worth a 38% reduction?
  4. The writing is dense and defensive, such as the excessive formalization in Section 4.2.
  5. Texts overflowed in Figure 1.
Reviewer_nEzqOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how to re-run a later step in a long data-analysis session, such as a notebook or LLM agent trace, using only the earlier cells that step depends on, while still getting the same output. The proposed method, StateLedger, tracks which version of each variable every cell reads and writes. Starting from the target cell, it recursively collects the cells that produced the needed state, similar to a backward program slice at the cell level. On 455 CoCoNote notebook targets, it exactly reproduces 342 outputs using 38% less code than the full history. It beats BM25, performs similarly to a reaching-definition slicing baseline, and does worse than a dynamic, execution-based baseline. The authors also run smaller experiments where an LLM extracts the state events from the session or writes the target code itself.

Critical strengths

S1: Comprehensive evaluation. The paper defines a clear, checkable metric: exact replay of the target's output from reduced history. It reports paired comparisons with clustered confidence intervals. It also openly reports unfavorable results, such as the null result against qualified slicing, the loss to the dynamic baseline, and no advantage in the fully online setting. This rigor makes the results trustworthy.

S2: A clean ablation of the core mechanism. The direct-producer vs. recursive-closure comparison (25/455 vs. 339/455) holds everything else fixed. It clearly shows that recovering indirect prerequisites is necessary for replay.

Critical weaknesses

W1: Limited novelty. At its core, the method is a backward, transitive, cell-level slice over reaching definitions. The paper itself calls it closely related to cell-granular reaching-definition analysis (Sec. 2). The experimental results reflect this: (1) It is statistically indistinguishable from a qualified reaching slice (342 vs. 323/455; the notebook-level interval includes zero). (2) The qualified slice also uses slightly fewer characters (1,078.5 vs. 1,097.7). (3) It is clearly worse than a dynamic slicing baseline (372/455). (4) The ablation's main finding, that transitive closure is needed, is the defining property of a slice, not a new insight. (5) The large gains are only over BM25 and a "plain static slicing" baseline that scores 46/455, which looks too weak to be meaningful.

W2: Weak connection to LLMs and agents. The headline results use deterministic AST analysis with the target code given; no LLM is involved. Where LLMs do enter, the method shows no advantage.

On LongDS with LLM-extracted events, it falls below both qualified slicing and BM25 (106 vs. 109 and 108/110). Without error repair it reaches only 50/110, and it needs repair far more often than qualified slicing (54 vs. 10 targets). In the fully online agent setting it ties BM25 (33 vs. 32/200), and both trail the full history (45/200). The one positive LLM result, teacher-forced generation, is compared only against BM25, not against the slicing baselines.

It is also unclear how much this matters in practice. Retained CoCoNote histories average about 1,800 characters, which fits easily in any LLM context, so saving about 700 characters is a small benefit. Longer histories, where reduction would matter, are mostly excluded because the full history fails to run (only 455 of 1,050 candidates kept).

W3. The presentation can be improved. The problem and key idea are difficult to grasp from the abstract and introduction. Much of the text goes to protocol details (guards, stability boundaries, intent-to-treat denominators). The formalism (lattices, least fixed points) is heavy for what is essentially a graph traversal. An early running example and a plain statement of how the method differs from existing notebook slicers would help a lot.

Questions for authors

Q1. What does StateLedger do that a standard cell-level backward slice with variable versioning (e.g., NBSafety/nbslicer-style analysis) does not? Could you characterize the 19 CoCoNote targets where it beats the qualified slice, and the cases where it loses?

Q2. How exactly is "plain static slicing" implemented, and is it transitive? A score of 46/455 suggests it may not be a reasonable baseline.

Q3. In the teacher-forced generation experiment, how does StateLedger compare against the qualified slice and the dynamic baseline, not just BM25?

Q4. Is there a setting with long histories where the method gives a clear practical benefit over both the full history and existing slicers?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.